In this edition of ‘Thoughts on Verification’, Verilab consultant Alex Melikian discusses Agile techniques and methodologies with Verilab senior consultant Bryan Morris. Before turning towards the verification world, Bryan came from a long history of software engineering and related project management. His experience and pedigree offer in-depth knowledge of how Agile can help development teams improve productivity and responsiveness when facing the increasing demands of a modern day ASIC/FPGA project.

In Part 1, Bryan explains the concepts and origins of Agile, as well as describing examples of how it can be applied in hardware development projects.

Special Note: For those attending Microprocessor Test and Verification (MTV 2012) conference in Austin TX, don’t miss Bryan’s presentation “Yes we Kanban!” on December 10th. Bryan will present the concepts of Kanban, an Agile methodology, and how it can work for your verification project management needs.

Alex Melikian: Hi Bryan! Thank you for joining me on this edition of ‘Thought’s on Verification’. Today we’re going to be talking about a topic that is a bit of a mystery for me and I suspect a few of our readers: Agile methodologies and techniques. Before we get into it, I would like to ask you to give a little introduction about yourself for our readers.

Bryan Morris: Great! I’m Bryan Morris and Senior Consultant at Verilab. I’ve been in the industry for about 27 years, the first 15 years were principally in embedded software design. Doing software on routers, wireless base-stations, and then gradually I moved up the food chain into a systems analysis role, where I was managing a group that did performance analysis of algorithms that were going to be implemented on ASICs. That led me into the ASIC design and verification space. Over the last 12 years I’ve specialized in ASIC/FPGA verification.

AM: How were you introduced to Agile? Or, in a nutshell how would you explain what Agile is?

BM: My introduction to Agile is interesting. Old things become new again, you know, the idea of doing incremental development was done back when I started in software. There were quite a few “agile” ideas that were being used: ‘evolutionary prototyping’, ‘incremental development’, that form a part of what Agile is today.

Onto my definition of what Agile means: Agile is actually a couple of things to me.
First, I think it means what you’d expect, it’s a process that can adapt to change more easily. I think it does that creating (maybe enforcing) opportunities for teams to communicate. That is, to truly understand exactly what they need to do, and also give feedback on what would work better as they progress through the project. The opportunities to communicate happen throughout the project as new facts or issues arise.

Secondly, agile for me is about focus. When you’re working on a project, the most productive way is for the entire team to work on it in a very focused way, and where the team clearly understands and shares the current project’s goals.

Agile also means trust. As a manager you’re going define the team’s current focus, give the team what they need to understand exactly what that is, then you are going to allow the team to define how they are going to deliver, give them the tools they need to do the job and then trust that they deliver. And also provide these opportunities for discussion and feedback. Then start the process again and again in an incremental way until the project completes.

A goal of agile is also to chunk your work up into little pieces that are manageable and where you can see the end goal. And then when you get to that end goal, you can review and discuss how well you did, and then work on the next goal, and then review, and so on until you complete the project.

AM: OK. That’s quite a comprehensive explanation, it definitely clarifies what Agile is for me. Of course, we can go on much longer if we wanted a thorough explanation of Agile. However, I would invite our readers who want to understand more about Agile to start with the Wikipedia definition page or even the Agile Manifesto website, which I believe is the original website of the Agile software development philosophy. Now, going back to the description you gave, Agile sounds more of a human management, or team management methodology.

BM: Exactly.

AM: So why is it relevant in hardware design? Why would Agile be interesting for hardware engineers?

BM: Well, let me go back just a bit to clarify or expand on what you just described. So a lot of the existing project management methodologies out there are command and control. The idea being that you would lay out something like ‘here’s the schedule everyone is going to work on the next twelve months’. That’s the command side. Then the control side comes in, and you track that project and make sure everything is working. Everybody is doing their job they are predicted to do. And as things come along, you would react and would update your schedule as appropriate. So that’s the control side.

What came out of that is that it works very well in fields that are predictable, and of low risk. For example, let’s use a construction project, most likely you know exactly how long something is going to take, because it’s an activity you’ve done a lot in the past. And you know that the skills are interchangeable of the people that you have. You know the resources you need ahead of time. So all of these are fairly predictable. There are always these risks that are unplanned, and so you react and adapt to those. In a command and control structure, in a predictable field, you can fairly accurately predict how much it’s going to cost, and how long it’s going to take.

The problem in software and hardware development, is that a lot of the projects that we do are dealing with elements that are new. They are bleeding edge. They have many technical risks. Usually you are resource constrained, you don’t have enough people, you don’t have enough compute power, not enough licenses for tools, etc. There’s just a whole host of things that can happen. And then there’s the technical risk factors. In most cases you’re exploring new territory, trying to build something that’s never been done before, or using a new tool or methodology. So you’re inevitably going to run into problems, you’re going to have to react and adapt to. In my view, the idea of trying to plan a 12 or 24 month project, and get it remotely accurate, is probably very slim. It simply is not going to happen, it’s nonsensical in fact.

So why Agile is particularly relevant to the hardware field is because we have all these risks, because we have all these unknowns, we always have to adapt to the changing set of facts that are coming in, or new issues that are popping up. They always come up, and you’re going to have to adapt to them. Planning in Agile starts with accepting that we can’t plan out for the next 18 months, and let’s just scale back and say well, what if we plan out for the next 2 or 4 weeks, and focus on a subset of functionality that we need to do, and we’ll start with features that have the highest value in the project. Let’s do the most important stuff first. Then we’ll repeat that that small iteration again and again until we’re done. Now your project is chunked up into many smaller projects. Instead of one big honkin’ project, you got multiple little projects that you’re going to deal with on an iterative basis. And that gives the team that focus, it gives the team intermediate goals that they can reliably, predictably achieve. Along with the ability for the team to react to adapt in the next iteration.

That’s why I think it’s relevant to hardware development, for the same reasons that it’s been successful in software. We’re doing stuff that we’ve never done before, we’re going to have issues where we’re going to need to create a management system, that is adaptable, and can react as quickly as possible to the changing climate or the changing facts as the project proceeds. Does that makes sense?

AM: Oh, absolutely … Are the elements you just described the reasons why those trying to put together a schedule and predict completion of hardware development projects often overly-optimistic and hence very inaccurate with their predictions?

BM: Yeah, I think it is an absolutely difficult problem to solve, because of all the things I just described. You know, I’ve done project plans in a very similar manner, to try predict when we’re going to be done, and it’s a hard, hard job. It’s not that we’re not smart enough, or experienced enough to accurately predict, it’s just that all these factors that come into play, and make it very difficult to create a reliable, tractable plan.

AM: It feels like coming up with a schedule that is accurate is like trying to predict the stock market. There are so many variables and risks, as you mentioned, that trying to predict everything down to a ‘t’ becomes more of a game of luck or a lottery rather than something that is more methodical and calculated.

BM: … and another thing is coming into play is, that the hardware teams are growing, they’re scaling, in the same way I saw software teams scale 20 years ago. Instead of a 5 person team, you now have teams of 20 or 30 engineers, or in some cases there’s even hundreds of people, that are all working on the same project. And that scale, and if you think of these people as points on a graph with all these interconnected points, this complex web of people you have to talk to and understand what everyone else is doing. Just the scale that we are approaching, makes it untenable to believe that you can actually manage that in a big picture side of things.

AM: That’s a good point. In historical terms, the expansion of complexity arrived in the software world first, and we’ve seeing this expansion of complexity in hardware catch up in the past years. I guess the paradigms, thinking and related methodologies of software project management of 10 or 15 years ago is starting to be applicable to the hardware project management of today.

BM: I believe so, yeah.

AM: So can you gives us an example of an Agile technique or methodology that helped you recently in a hardware related project?

BM: Yeah, there’s two methodologies that are particularly relevant to hardware. It all depends on what kind of culture have existing in your team. I’ll describe them very briefly … they have some similarities. The first one is called ‘Scrum’, which is probably one of the more well known Agile methodologies. Scrum is basically what I described earlier, where you chunk up your project into mini-projects, or iterations, or in Scrum terminology a “Sprint”.

These Sprints can be any size that is meaningful for your team from a week to four weeks. It defeats the purpose if you make it too long. Software usually work in one or two week iterations, but I think in hardware that two weeks is a minimum, with a better iteration duration of 3 or four weeks.

But lets keep it simple and say you’re going to chunk your project into 2 week sprints. And at the beginning of two weeks, you do a Scrum Planning meeting. The goal of this meeting is to allows the customer, or your customer’s delegate to define for the team exactly what is the set of next highest priority features that need to be developed. A customer’s delegate could be someone in your organization that is responsible for knowing what the customer wants, this could include the product management or marketing department.

The prioritization should be what’s the highest value to the customer that they would want to see. At the beginning of the project, you would prioritize the basic or must-have functionality that is going to be created.

During that Sprint Planning session the team gets to ask questions and understand fully what those feature’s behaviour should be — this is an opportunity to create a meaningful discussion. The team is responsible for accepting, and planning out the next two weeks; they’ll define their delivery goals, and how these goals are going to be achieved. During that time they know exactly what the goal is because it’s only 2 weeks away. They also know exactly what customer’s acceptance criteria is going to be, because the customer is not only going to define the feature list that needs to be worked on, they are going to define what it means for the feature to be complete. So the Scrum Planning session, focuses the team’s objectives over the next two weeks.

Then on a daily basis over the next two weeks we have Sprint Daily Stand-up meetings — which are very short. In fact, it should be limited to each person giving a 30-60 second synopsis of what they’re doing and any blocking issues preventing them from proceeding (the term ‘stand-up’ comes from the rule that everyone stands — encouraging the meeting to be short). Again to create an opportunity for discussion on things that are going well. What is particularly important though is for the team to discover the blocking issues as soon as they are known. But once you identify these issues to take them offline and figure out how to solve them. So at any time during the project, you only have a 24 hour point, where if you’re blocked, you can come to the daily stand-up and can say “I need help”. Here’s where the manager comes in, and instead of a command and control, their role is to facilitate the team and help remove those blocking issues.

The daily stand-up meetings happen until the end of the 2 week sprint. At the end of the two weeks you have what’s a called a Sprint Review session, or ‘demo to the customer’, where the customer comes back again, they see what you’ve produced. In hardware development this proof of completion can mean RTL + associated tests with coverage metrics with some pre-defined acceptance goals. The customer’s role is to accept it or not.

The last piece of the puzzle is the Sprint Retrospective which happens as the last meeting in the current sprint. where the team gets together to discuss what went well in the Sprint that they just completed, what didn’t go so well, what they want to change or keep the same. With action items to be considered as inputs into the next iterations Sprint Planning session. This is like a project ‘post-mortem’ that many teams do at the end of project, although the scope is on the work done during the sprint.

And that’s how the Scrum process works, where you do two week cycles, at the start the customer brings in the next higher priority features and the team implements those features and proves they are complete. So at the end of those 2 weeks you have a set of features that are complete and in terms of what the customer expects. So that in a nutshell is Scrum.

Kanban is another agile process where instead of an iteration it is continuous process. It’s main focus is on visualizing your process. For example, many teams use a whiteboard with a set of columns that define each step in your process. Tasks move across the board horizontally from left to right. The columns describing each step in your process could be as simple as ‘analysis’, ‘design’, ‘implementation’, ‘testing’ and ‘accepted testing’. And so tasks or features flow from one side to the other side column by column. And it’s a continuous process, meaning that whenever the team is ready to start a new task, the customer has an opportunity to come in and say here’s what the next highest value task or feature to do. It can then be slotted into the first step (or column) of the process. Being continuous the team can also can talk about how they can optimize their process to improve their productivity. Compare this to the Sprint Retrospective that is done at the end of each iteration.

An interesting thing about Kanban, is firstly, it visualizes what the team is working on at any time, but it also quickly identifies the blocking issues when a particular task stays in a column too long. Since you also assign a ‘maximum number of tasks’ on a per-column basis it blocks all the other tasks further behind upstream. There’s a lot more to Kanban with a lot more empirical metrics that can be gathered as you’re going along. These metrics capture how fast you’re moving tasks across the board, and how long things take to complete.

In fact, there are a lot of metrics involved in both Scrum and Kanban that allow you to see your process with the goal to improve your process.

So those are the two main agile frameworks that I think are relevant to hardware development. Picking which to use is very dependent on the culture you have. For example, if you have a more defined set of features to build then the iteration-based development in Scrum is definitively good introduction to an Agile methodology because it lets people set short term goals and achieve them. Kanban is better suited if you’re in a situation where things are less defined, or changing more rapidly. It allows you opportunities to adapt a lot faster because it is a continuous process.

This entry was posted by Alex Melikian on Wednesday, November 28, 2012 at 6:16 am and is filed under Interview, News. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.