The rhythm master – ITNEXT

Juntao Qiu

All great teams are alike; each poor-formed team has their own problem(s)

If you walk into any well-running agile team, you would surprisingly find that what people do are pretty simple and straightforward on daily basis. It’s not saying the problems they’ve been solving are less challenging than any other projects. The tasks are just simple and well defined and have less friction with each other or with their environment.

They delivery workable software, in an end-to-end fashion, every piece of work has clearly defined acceptance criteria, tasks have well-defined priority, the backlog is sufficient but not creating any anxieties, maybe you would even find the team is full of friendly and funny people too. It’s just perfect.

Furthermore, whenever they start something, they begin with a simple task as a prototype, once the prototype is finished then they make some improvement based on it gradually, in the incremental fashion we all appreciated. They refactor the code had already written, sometimes even rewrite part of it, but people have the confidence to do so because of the underlying tests. Overall code quality from the outside perspective is ensured by just a sufficient amount of automation tests.

Every perfect team like this are not come easily by any means. On the contrary, you have to try your best to reach a stage like this. Fortunately, there is a path.

However, if you ask the same question to anyone in that team: “What’s the essential principle do you think in agile?”, You’ll surely be surprised by the variation of answers you got. Someone believes the extreme extent of engineering practice would lead the team to success, like pairing programming, TDD, or CI/CD. Also, others might think the communication methods they are adopting like iteration plan, storyboard, regular business showcase are the essential things. Some might argue that in a more higher level as embracing the changes, fast response time is vital. Of course, all those answers make sense in some way, people tend to conclude and form their own view as they perform the daily job. While in my opinion, agile, or maybe lean as well, the fundamental can be described as ‘incremental improvement.’

By saying Incremental improvement I mean: start from the highest priority tasks that have well understood and relatively clear acceptance criteria and enhance based on those tasks. During the gradual improvement, also build the rhythm of delivery and ritual and maybe even the culture that fit the team. Once you have all this built, the rest then just emerge smoothly. And that’s exactly what will be discussed in this article: how to make the incremental improvement practically possible, especially when there are many constrictions around in the real world projects.

To achieve incremental improvement, you have to put much effort in advance. Such as

  • Requirement break down — elaboration
  • Applying INVEST correctly
  • Visualise the process
  • Build the culture (value-based delivery approach)

Those extra efforts need a Business Analyst as the driver to make that happen. Different ways of splitting requirement and even the rhythm of releasing the backlog into ready-for-dev could lead a totally different result. But before we dive into the details, let’s firstly see how important the rhythm of delivery could be in any agile teams.

In 1975, the famous psychologist Mihaly Csikszentmihalyi discussed the concept of flow and started to research it as a serious science topic in the first time, although people from different backgrounds have already found and experienced that for quite a long time.

In positive psychology, a flow state, also known colloquially as being in the zone, is the mental state of operation in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity. In essence, flow is characterized by complete absorption in what one does, and a resulting loss in one’s sense of space and time.

To enter the flow, Csikszentmihalyi suggested that you have to do at least 3 things first

  • A clear vision or target
  • Accurate and valuable feedback
  • The balance between your capability and challenge (both in a relatively higher level)

That means if you want to work in your possible most effective pace, you should define the target clearly, and when you try to achieve that goal, you need continuous feedback from the outside world. And your capability or skills for finish the task need to match the challenge of the task, if the challenge is far beyond your capability, you would soon turn to frustrate while if it’s too easy for you to do that, you would quickly get bored.

The flow can not only maximum individual’s potential but also can be applied to the team level. When it enters a flow state or has a rhythm established, team members tend to be more effective and cooperate smoothly, whatever the task you hand over to them, they can finish it with no problems.

Since it’s crucial to establish a rhythm, and Business Analyst plays a vital role along with this build process, the question comes: how can we achieve it in the real world? Let’s get started with a typical scenario.

A Business Analyst in the team is just like a voltage regulator in any circuits. They act as the bridge between the external environment and the team. On the one hand, they have to manage the expectations from any channels outside, analyse them and prioritise carefully. On the other hand, they have to trade off different factors under certain circumstances and protect the team from overwhelming tasks.

In some extent, I would say it’s more important to protect the team than satisfy stakeholder because the team is actually the people who generate valuable outcomes.

In the worst cases, Business Analyst just literally passes the requirement from upstream to the delivery team without deep analysis. I have seen in some scenarios, the Business Analyst just like a parrot and refused to analysis the raw requirement, and in the kick-off then it’s hard to answer questions asked by Devs or QAs. As a developer, I would refuse to write the code if I’m not convinced by the business value (if any) in a story.

Just applying the traditional tools like INVEST (maybe together with SMART) correctly when split requirements into chunks rationally could ensure the team in the right direction. Of course, you might have to use other tools in different scenarios to make the right splitting.

Questions are:

  • What’s the right way to do INVEST, if there is a right way?
  • Even you used it correctly, it might be still not working, what would you do then?

For the right way to do INVEST – includes how big of each chunk of work is fit, typical patterns of split such as workflow, business rules variations – there are many articles already out there, I’m not planning to describe those here. However, I’m particularly focused on the second question, that is, you have already mastered how to apply INVEST in different scenarios appropriately, but the way you split story doesn’t fit the team capability structure(or the way the team works), what should you do then?

In the real world, you might find there are many variations about the team capability models. Conway’s law is everywhere, organisations have different communication structures, which lead to different team structures. And sometimes it impacts even how you would split a requirement into user stories.

At least you could see two of those variations: frontend and backend separation and end-to-end separation.

From time to time, the agile practices you read from books, hear from the conference just not work in your project. Suck as vertical split (cake metaphor maybe), end-to-end delivery when you try to apply it. Occasionally you will find the real world is different. Chances are before you even start writing any code, the final version of design has already been demonstrated to the most important stakeholders, and even the colour scheme, typography is finalised.

Although you have regular showcases during the development, it typically not works well, the feedback doesn’t provide much value to guide the enhancement. Everything could change after the final showcase, and I guess we all experience that enough.

In scenarios like this, it seems separate front-end and backend could speed up, and make more sense practically. The dedicated UI Dev can easily convert the Hi-Fi into HTML/CSS/JS, and in the surface, things like performance, system resilience, high concurrency should not be involved into the UI work at all. In fact, many projects are just running in this manner, to me, it works well for fixed scoped project.

Unfortunately, most of the projects I’ve been worked are dynamic. There are obviously some drawbacks of this approach:

  • To not be blocked by the backend, frontend dev would set up a mock server
  • Backend need some mechanism to make sure the contract matches between frontend and backend
  • Additional test suite required to be set up
  • The real integration is delayed
  • The feature could not be delivered end-to-end (you have to wait to the slowest end to finish)

And it’s highly likely that the front-end is waiting for the back-end to finish the API or vice verse. Also, those hard to track status could lead intransparent, and then finger pointing, management cost, internal frictions, and so on. Even we could establish sophisticated engineering practices like mock server, contract tests, the process is still painful and error-prone. It is even worse to only demonstrate the fake / mock version of UI that doesn’t actually talk to any downstream services, that make the team and client both lack of confidence.

In contrast, an alternative is vertical split and delivery features end to end, with both front-end and backend, and then sign off as a whole. Of course, the main blocker of this approach is the team’s capability, and the gap turns to be even wider since both ends becoming more and more complicated (from the programming language, tools to the programming paradigm, and so on). In the vertical splitting, we need developer has both the skill set required for frontend and backend, or at least pair with one another to finish a single story.

Additionally, that manner has its own problems need to take care of:

  • How the skilled UI dev and backend dev work together effectively
  • An end-to-end user story might take longer to be delivered
  • A developer needs more time to speed up / skill up

In first glance, those two different approaches are pretty contradicted and cannot work together at all. In most cases, if you just look at the outcome of each method, they are the same: finish the requirement right on time, with reasonable quality build in. However, something is missing in the front-back-end separation approach: the developer’s experience.

I hate the feeling that I’m not sure if something moved to DONE is actually done or not. I hate to be only part of the whole system when you have your dependencies to others and at the same time be depended on someone else in the ecosystem. No one knows if the feature you’ve been developing has its user or not, not to mention the business underlying value. So for me, I would try my best to implement each story end-to-end, from database/service to UI.

I think even in the extreme environment, we should adopt an end-to-end approach to delivery. Firstly, the team does not see a user story from a technical perspective anymore, and they divide it from the feature (or unit with business value). A feature could be finished by using different technicals in the delivery phase, the functionality should be atomic, cannot be split into front-end and backend. Secondly, if we inspect another factor that is most important to the success of any project: people’s capability and growth, end-to-end then has much more advantages. After a project, frontend dev might have mastered how to use terraform to set up infrastructure and do the application maintenance on cloud. In the meanwhile, backend devs understand how the JavaScript in frontend ecosystem had evolved recently.

Moreover, it just requires a tiny adjustment that can improve the overall experience greatly during the delivery process. Even in some extreme cases, when the scope is defined fixed at the beginning of the project. In fact, by just applying INVEST correctly can help the team to establish the smooth, stable and effective delivery rhythm.

INVEST is a fundamental part of achieving incremental improvement. Without fine granularity, independent user stories, gradually enhancement then because impossible. In fact, that requires consciously practice at many different levels. INVEST has its profound impact in agile: if the split if too small, then you likely introduced much more managing cost. If it’s too big, it’s difficult to define the boundary and hard to prioritise, or even worse, it could affect the overall atmosphere: no one want to see the endless card pinned on DEV column for a week.

Just like many other things in the real project, as a Business Analyst, split the story to a perfect size need many trade-offs from different dimensions, sometimes it’s more like an art.

Let’s take a look at some concrete examples here:

Let’s say, we need to build a jigsaw like application, it has a staffing history page for a particular user, that includes: navigation bar, basic information, staffing history, user profile, and so on.

Assume our first story is to display basic information (marked in the red rectangle) section on the page.

The story then should include: talk to backend service to fetch data and surface it in UI. While the data could be fetched from a real database or a simple JSON file, it depends on how the persistence strategy in the backend. As the first user story, it could have an associated tech task: setup the skeleton code. For example, use create-react-app to set up frontend, and use gradle and corresponding template to generate backend file structure, and so on.

As a valuable user story, it meets the following criteria:

  • Vertical split, the end to end implementation
  • UI align with the mockup

Although it’s relatively a simple and straightforward story, it helps developers to establish the feedback mechanism while they’re doing the job. Such as to what extent to do TDD, how to do Code Review, and experience kick off, desk checks as a team, etc. In other words, it’s not only a user story but a specific example of how you set up the feedback loop – the essential part of establishing the rhythm. Once people understand the idea of how to do the basic version, you can gradually put more detail and make it more complicated, and it turns to be more smooth then.

A reasonable enhancement of the first story could be:

  • When a user doesn’t exist, show some error message
  • If some field in the data is missing, fallback to something else

Since you have already had the first story played, the team would be more confident about both how to do it and what to do. Once people see something build from scratch, then it tends to be something appeal themselves to make it better. It’s actually very important to make people confident by small but meaningful achievement first, and then some enhancement follows.

For the more complicated feature, you can still achieve gradually improvement by applying INVEST. When signing off the basic version, just assume the final version didn’t exist at all. The enhanced version could be signed off separately.

As shown above, the content of the part marked as read rectangle is from another backend service, you have to integrate with some other product in a particular time, of course, they don’t necessarily have the same delivery rhythm as yours. Additionally, since you might do not have the priority yet, treat the whole section as a story (which a commonly seen pitfall) would make it hard to sign-off, and potentially would slow down the velocity the team has. A better way to do the split is that assume that part doesn’t exist at all (or out of scope), and then do the estimation, development, test based on that assumption.

In other words, even the Hi-Fi is ready before development, you should always split user stories based on business value, priority and other constrictions (say, the cost of integration), other than cut physically from page to page. In most cases, split by UI is a bad idea. On the one hand, business value is not present since the cut-off; on the other hand, the team cannot find the rhythm to produce (and that’s the most important part of delivering).

In the micro view, each well-split user story that has end-to-end value can trigger the flow. While in the larger scale, Business Analyst could introduce other practices to ensure team members could see the target clearly, and make sure the team is orienting in the right direction. Undoubtedly, visualisation tools could ensure that to happen.

There was a saying I read somewhere years ago that visualisation methods used in agile are utilising the sense of shame of people: when you see the card has already on the same column on the wall for a couple of days, you should feel some pressure from it. I believe that theory is pretty superficial, agile itself should be a methodology you can use in a highly trusting environment, people in this situation should feel something like: looks there is something went wrong? Maybe there are some technical challenges we didn’t aware of, or the story split if not done well, and let’s see what I can do. Once the problem is identified, it’s time to help to fix it and then learn from it, and that’s it.

There are definitely more things to pay attention to during the development by proper visualisation. From physical wall to iteration report, from burn-down chart to RAID, those tools can help the team significantly understand what the progress looks like, and any potential problems, and waste, blockers should be clearly identified, and so on. It’s sometimes critical to help team member to realise the gap between the target and current state, and if the tool can provide how to close the gap that would be even better.

Developers can easily get lost in technical details and forget where they are in a project. A bright and well-structured iteration report can show well to the team all the overall situation during the iteration. The achievements: requirements that be signed off, time spend, cycle time of each story, important business value delivered, etc. I also found it actually really helps to spend some time (not too much, 15 minutes – half an hour should be enough if you have a template and with help of tool like Jira) on generate a report, and send to team members and related stakeholder — that summary of a particular state and let people know are we there yet? and how far away is our goal? (Nicole had been done iteration report and different types of visualization very well, ask her for more details if you’re interested)

In one of the consulting project, I had been worked on help a delivery team to set up requirement splitting capability. In an INVEST 101 principle workshop, a developer asked me: apart of other principles, what do you mean by negotiable anyway?

One aspect of culture in that client is: you cannot challenge the requirement after analysis phase finished, you have to do whatever agreed in a particular stage and then plan all the resources (of course, developers are part of the resources, like project budget or a keyboard) required to meet the deadline. That’s a typical method adopted by many companies in the manufactory industry, and we know that it’s never work in the software industry. Anyway, the delivery team has no power to negotiate at all, and the deadline is even out of the delivery team’s knowledge. However, the consequence of missing the deadline would be, ironically, their responsibility.

Obviously, each team needs and has its own culture, that determines the way of work in the group. Also, the way of work will play back to the team to cultivate the people and then shape the culture again. Fortunately, most of the teams in ThoughtWorks, the culture grows from the ground, and no one can force anyone to apply a particular culture.

In fact, a simple brainstorm could highlight the team’s value, work atmosphere expected, the way of work, and so on. It was something we did once in a Team Value Session.

The most stunning thing about culture is you can basically put anything in, and the different person thinks their point is most important. Although I cannot judge what is essential or isn’t to a team, I want to emphasise that some points can help to build a flow that impact delivery rhythm.

  • Learn from failure
  • Measure the gap between the target and real
  • Feedback
  • Be professional

Small but complete achievement is a simple way to incubate a health culture. Of course, culture is not something you say, but something you really do. For example, set up a fixed deadline for each user story — you have already described it as ‘failure is not acceptable’, even you say that’s all right to make mistakes, people won’t believe you. On the other hand, clear acceptance criteria, requirements marked as an out-of-scope, definition of done for each task demonstrating how you would measure the gap between the target and reality. Moreover, feedback, of course, is always the essential element in any agile practices (Pair, Code review, Standup, Retrospective, etc.).

A Business Analyst is actually the one who controls the whole delivery rhythm in a team, he/she connects outside and inside, manages the expectation from both client and team members. To the team, they have to split requirement to well granulate stories, help to build appropriate mechanism for feedback. To the stakeholders outside the team, they have to manage their expectations, facilitate a lot of activities: visualise blockers across teams and drive to trace and remove those blockers, analyse requirement, prioritise requirements based on business value. Also, keep outputting them as deliverable units to the team back continuously.

Even in the project that has a relatively fixed scope, Business Analyst can build the pipeline of user story through appropriate granularity splitting by using tools like INVEST. Additionally, by utilising various of visualising tools, they can build a stable and effective feedback channel. During implementing, Business Analyst could lead by example to demonstrate and emphasis learn-from-failure other than following plans, start from small but solid prototypes, improve piece by piece. And in the bigger scale, Business Analyst could make sure the whole delivery process is running under a safe, transparency, ordered, and smooth atmosphere.