|
| |
Case
Study: Software Company in Networking Market
[Company name withheld upon
request]
Testimonial:
This
client develops complex, highly distributed, server/client/web-based software
used to manage complex data networks and network-based applications for
enterprise and service-provider customers.
Their software enables customers to identify and correct problems before
they seriously impact their highly critical network operations.
On-time delivery of product releases and high product quality are
essential.
“Tom Dennis was instrumental in assessing the engineering organization, and
in recommending and implementing changes associated with the enforcement of our
development methodology, the consolidation of QA into a separate and co-equal
organization within engineering, and the consolidation of all Release
Engineering activities into a single group.
He also recommended effective changes in the way these organizations
operated. The net result of the
implementation of his recommendations was on-time delivery of product releases,
high quality products with minimal customer reported issues, and motivated
employees. His efforts helped to
significantly reduce development and product support costs, and contributed to
increased product revenues.”
– Chris Crowell, Vice President of
Engineering
Case Study:
Background:
This $60M software company in the networking market derived about equal revenues
from sales to new and existing customers, and from support and maintenance
contracts from existing customers. The
biggest engineering-related issues facing this company were delays in the
delivery of new software releases (both major and minor releases), and quality
issues reported by customers.
Problems Identified:
This 250-person engineering organization was lacking the
enforcement of discipline of a solid development methodology.
Further, it had the critical functions of test- and release- related
activities spread out among five departments. This decentralized approach gave
no one the overall responsibility for these two critical functions, and this led
to lack of ownership and finger pointing within the organization.
That development delays and quality issues were present was not
surprising.
Actions Taken:
1) Methodology Enforcement: Forced adherence to the formal
development methodology was mandated. This
meant that design specs were to be written, reviewed, and signed off on before
coding began. It meant that
independent code reviews were to be completed before code submission to the
source code control system. It
meant that software development tools, such as tools to identify and correct
runtime errors, to identify and correct memory access and memory leak errors, to
analyze code coverage (whether or not code has been executed and tested), to
analyze software performance and eliminate performance bottlenecks, etc., were
run on all code before code submission to the source code control system.
It meant that thorough unit testing was to be completed before code
submission to the source code control system.
It meant that independent code reviews on all code changes to correct
bugs were to be completed before submission to the source code control system.
It meant that all other elements of the software development methodology
were to be complied with. Metrics
of compliance with this were to be maintained and reviewed.
2) Test Organization: Five disparate test groups spread among the
development groups (integration test, core functionality test, applications
test, device management test, and performance test) were consolidated into one department of
four groups (core functionality test, applications test, device management test,
and performance test),
independent of the development groups, each with a manager, and all reporting
into one director who was at a peer level with the development directors.
This consolidated test organization was given the charter, for each new
release, of concentrating initially on working very closely with development on
the integration testing of portions of new releases, and when ready, moving to
an independent software quality assurance (SQA) activity of the complete release
where tests were to be conducted more independently of development interaction.
This department was to perform integration testing, functional testing,
system testing, white-box testing, black-box testing, negative testing,
performance testing, and regression testing (to ensure bugs have been properly
fixed). They were to be the
internal proxy for customers regarding the quality of the releases they were
testing. It was the test
department’s responsibility to make the determination of whether or not a
release was ready to go to customers.
3) Release Engineering: All functions related to release of the
product were consolidated into one group called Release Engineering, under a
strong manager. This included
responsibilities for release coordination, release project management, source
code control system and tools, build engineering and build tools, bug tracking
and correction system and tools, daily bug tracking and correction assignment
meetings, weekly project review meetings, sustaining engineering coordination,
and regular interaction with all functional areas in the company that had an
affect on or were affected by the release of new products (included development,
test, usability engineering, performance engineering, technical documentation,
customer support, product management, and marketing). The Release Engineering group had the responsibility for all
releases of product to the field, both new releases and sustaining engineering
releases (patches and maintenance updates).
This group worked with all of the other groups in determining the
schedules for all releases, and for tracking those schedules to ensure on-time
delivery per committed schedules. This
group also instituted significant release process changes, including to the bug
review process, to the weekly project reviews, etc.
Results:
1) Methodology Enforcement – The result of enforcing adherence
to the development methodology was higher quality initial code, reduced testing
time by the test group due to fewer bugs being found, delivery of quality code
in shorter timeframes, and fewer customer problems being reported.
The overall economics of this is summarized below.
2) Test Organization – By combining the test groups under one
department at a peer level with the development departments, the testers were
given a stronger sense of purpose and co-equal status with development.
It also gave the testers greater freedom and testing variety to reduce what often had become tiresome and demoralizing repetitious
drudgework. It enabled the
elimination of duplicative testing (and unneeded people who were doing
duplicative testing), the identification of marginal performers (managers,
project leaders, testers), and enabled the groups to pinpoint key quality
problems to root causes in development so that quality could be designed in
rather than tested in. This
resulted in significantly improved quality.
The overall economics of this is summarized below.
3) Release Engineering – By concentrating all
release-related activities under one group, significant improvements in
communications and coordination among groups resulted.
It not only gave those in Release Engineering the focus to concentrate
on the specific tasks necessary to get good quality product delivered on time,
but also provided the other groups (development, test, documentation, product
management, etc.) a common point of interaction to work with.
Projects moved from missing delivery dates by 3-6 months on 12-18 month
projects to delivering on time.
Overall Economics:
The net result of the changes implemented resulted in about $4M in
development cost savings, and in at least $5M in additional revenues.
This is explained below.
(1) Development Costs Savings of $4M: This was determined
as follows: (a) People: The organization as a whole was able to complete a
project with 10 fewer people by enforcing the process and therefore not needing
to do things over, by eliminating redundant efforts, and by keeping everyone
fully informed along the way. With
an average loaded salary of $150K/person/year, this provided $1.5M in
development costs savings. (b) Time: The organization was able to deliver higher
quality product on time, saving 3-6 months.
Assuming a 4-month time savings on a 1-year long, 50-person project, the
net savings was $2.5M in development costs.
Thus the net development cost savings was $4M.
(2) Revenue Improvement of $5M: Whenever a project is late,
revenue that would have been made from the time the project should have been
delivered until it is actually delivered is lost forever.
For this $60M company, $30M was from sales and $30M was from maintenance.
In this company, one-half of the revenues came from new releases, or
$15M/year from new releases. If a
release is delayed by 4 months (one-third of a year), then $5M in revenue is lost.
By delivering on time, that $5M can be reclaimed.
In fact, the revenue gains will actually be greater, since sales will not be
reduced due to late delivery, and since the higher quality of the product will also improve sales.
| |
|
