Leaving patterns & practices

[J.D. Meier's Blog]“Life is like skiing.  Just like skiing, the goal is not to get to the bottom of the hill. It’s to have a bunch of good runs before the sun sets.” – Seth Godin

It's been a good run.  After more than 10 years in patterns & practices, I'm on to my next adventure here at Microsoft.

For this post, I wanted to take a stroll down memory lane.  During my time at patterns & practices, I learned more about project management, application architecture, software development, processes, teamwork, leadership, product success, and making impact than I think I could have ever hoped to learn anywhere else.  Perhaps the best part, is that I have 10+ years ofdeliberate practice in the art of “best practices” and I’ve had a chance to work with some of the most amazing people in the industry, lead SWAT teams around the world on fantastic voyages and epic adventures, and leave a legacy of several high-impact Microsoft Blue Books, all while mastering the art of prescriptive guidance.

There are a few ways I can share my journey in patterns & practices.  One of my managers referred to me as “the abilities PM” because of my focus on quality attributes (security, performance, scalability, etc.)  and that’s a pretty good place to start.  My role as a PM (Program Manager) in patterns & practices, can largely be defined in terms of my impact on these three areas: security, performance, and application architecture.  (Yes, there is more to the story, but those three areas, serve as a good enough lens for now.)

The Security Story    
It all started when I joined patterns & practices (PAG or the Prescriptive Architecture Guidance team at the time.)

I moved from Microsoft Developer Support with the expectation that I would share and scale broadly with customers what I learned from living in across the hall from folks likeScott Guthrie, Brad Abrams, and other .NET Framework leaders.  Having performed more than 650 customer architecture and design reviews, I was in a unique position to share all the principles, patterns, practices, and anti-patterns that I had seen across a large number of customer projects, many of which were on the leading and bleeding edge of this space.

But life has a way of throwing curve balls.  Around this time, security was Jim Allchin’s top priority, and he asked patterns & practices what we were going to do about security.  In turn, my manager asked me, what I was going to do about security.  Coming from a middleware background, security was something I was usually trying to work around, especially when I had to flow callers to the backend.  My new challenge was to design security architectures for our application platform on .NET.  Boy, was I in for a ride.

I knew security was the name of the game, but I had a lot to learn in terms of leading project teams around the world, brokering in all the right parts of the company, our community, and the industry.  I also had a lot to learn in terms of how to create prescriptive guidance.  The purpose was not to just put out a book.  The purpose was to drive customer success on the platform.  This included creating a durable, and evolvable security story that our customers could build on, and that we could use to “train the trainers.”  This also meant creating a knowledge base that we could use as an anchor fordriving product feedback and platform change.  This was all in addition to learning how to think about security from an architecture and design standpoint, in a way that could help customers build more secure applications.

And thus the first Microsoft Blue Book, Building Secure ASP.NET Applications, was born.  It was our answer to IBM Redbooks.

While I learned a lot from doing end-to-end security architectures and putting our variety of products and technologies together, the real learning came from driving Improving Web Application Security, or “Threats and Countermeasures”, as we affectionately called it.  This journey was about learning how to bridge the gap between architecture and design, development, test, and operations.  It was about learning how to think about security from a threats, attacks, vulnerabilities, and countermeasures perspective.  It was about thinking about security in a more holistic way, and our little mantra was “secure the network, host, and app.”

We had the ultimate security dream team, on mission impossible, and we pulled it off.  Our core engineering team included Alex Mackman, Michael Dunner, Srinath Vasireddy, Ray Escamilla and Anandha Murukan, and we had an extended team  of industry champs including Joel Scambray of the Hacking Exposed series.  (In fact, check out the extensive team list at the bottom of our Threats and Countermeasures Landing Page.)

How do I know we pulled it off?  We had outstanding results in our platform security competitive study, and we were told that is was our prescriptive guidance that made the difference in a very big way.  In addition, our guide was downloaded more than 800,000 times in the first six months, and it quickly established the mental models and language for how our growing customer base thought about security on the .NET platform.  It was a powerful thing when customers would say to us back in our language, “We’re using the trusted subsystem model  …”  It was like poetry in motion.

The big thing we learned from the journey was the power of having end-to-end application scenarios, along with maps of threats and countermeasures, while baking security into the life cycle, and using proven practices, like threat modeling, to significantly change the game.

Here are some of the key security deliverables at a glance from the various security adventures over the years:

• Windows Azure Security Notes

• Improving Web Services Security: Scenarios and Implementation Guidance for WCF

• Security Engineering Explained

• Threat Modeling Web Applications

• Security Guidance for Applications (How Tos, Guidelines, Checklists, etc.)

• Improving Web Application Security: Threats and Countermeasures

• Building Secure ASP.NET Applications

This doesn’t include the product feedback work we did, or the work we did to bake security into Visual Studio / MSF Agile, or the patents we filed, which were the icing on the cake.  As a sidenote, behind the scenes we called our “threats and countermeasures” guide “WOMA.”  It was short for weapon-of-mass adoption, because our field was continuously telling us stories of how they were winning accounts against the competition.

The Performance Story    
Performance was right up my alley, but it was a significant shift from security.

Ironically, I got a lot better at tackling security, by learning more about performance.  The big lesson I learned was that you have to bound or constrain your efforts in some way.  Just like performance can always be improved, so can security, so it’s all about prioritizing, knowing where to focus, and connecting back to the business objectives, aside from the technical perspective, and user experience.

Our first epic adventure in the performance space was the guide, Improving .NET Application Performance and Scalability (or, “Perf and Scale” as we affectionately called it.)   It was where I first got to work with folks like Rico Mariani, Jan Gray, and Gregor Noriskin.   It was mind blowing.

Working on performance and scalability was probably the most technically challenging exercise I’ve had at Microsoft.  I remember spending countless days and nights walking through CLR internals, figuring out how to optimize collections, sorting through multi-threading patterns, and mastering how garbage collection really worked.  Strangely enough, the “ah-ha” that I appreciated the most was figuring out that we could think of performance in terms of response time, throughput, and resource utilization (CPU, memory, network, and disk.)  That little lens was the key to figuring out how to do effective performance modeling and solving nasty performance bottlenecks.  It also helped us parse complicated performance scenarios down into bite-sized chunks.

Here are some of the key performance deliverables at a glance from the various performance adventures over the years:

• Improving .NET Application Performance and Scalability

• Performance Testing Guidance for Web Applications

While performance took me to the depth of things, it was application architecture that took me to the breadth of things …

Application Architecture    
Creating the Microsoft Application Architecture Guide was a chance to pull it all together.    

It was a chance to finally showcase how to put our platform technologies together into common application archetypes.  Rather than just a bunch of technologies, we could talk about our application platform very specifically in terms of application types: Web applications, RIA applications, desktop applications, Web services, and mobile applications.  We could talk about scenarios and trade-offs.  We could look at the shapes of applications in terms of architectural styles and patterns.  We could look at cross-cutting concerns, such as caching, communication, concurrency, configuration management, data access, exception management, logging, state management, validation, and workflow.  We could also walk the various quality attributes, like performance, security, reliability, manageability, usability, etc.

As you can imagine, this meant putting together a comprehensive map of our Microsoft application platform.  The surprise for me was that by putting together the map and looking broadly at the industry, it was easy to see the forest from the trees, as well as better understand and anticipate where the industry was going in terms of what was growing and what was dying.  It was a great way to get ahead of the curve.  In fact, I’ve continued to map out our Microsoft application platform each year, as a way to quickly see the forest from the trees and to figure out where to spend my time.

During my adventures with application architecture, I got to learn a lot and work with amazing people.   I also learned how to go across a lot of information faster and easier, and bring teams of people along the journey.  The secret was to keep creating maps that helped everybody get on to the same page fast.  This was an invaluable approach as our team was hunting and gathering all the pattern collections we could find.  We basically built a large catalog and constellation of application patterns in the form of maps.  While we didn’t include our maps in the guide, they helped our team and extended team ramp up in various spaces very quickly, as well as advance the practice of application architecture.  Basically, we could browse patterns of solutions at a glance.

If you read nothing else, check out A Technique for Architecture and Design.  Behind the scenes, we created this technique by synthesizing the expertise of more than 30 top solution architects, that had years of experience with structuring and designing end-to-end applications, dealing with security, performance, and reliability issues, and dealing with cross-cutting concerns.  The idea was to put down on paper, a proven practice for rapidly modeling applications on a whiteboard and identify risks earlier vs. later. 

What’s Next?    
I’ll have more to share as I go along.   What you’ll most likely see is a shift to more focus on strategy, execution, and business impact.  I’ll also share more information on the art and science of program management.   I’ve been mentoring a lot of people and I think the PM role at Microsoft is a very special one.  One of my mail goals is to broadly share the lessons I’ve learned from driving projects and leading teams and making impact as a PM on the patterns & practices team.

patterns & practices Guidance at a Glance    
Meanwhile, here is my catalog of patterns & practices guidance at a glance.  Note that a lot of the prescriptive guidance I’ve worked on is out of date because of changes in technology.  That said, you can still learn many of the key principles, patterns, and practices that  the guidance is based on.  In this respect, much of the guidance is “evergreen” in that it’s timeless and durable.

Books

• Application Architecture Guide 2.0 (CodePlex)

• Building Secure ASP.NET Applications

• Improving .NET Application Performance and Scalability

• Improving Web Application Security: Threats and Countermeasures

• Improving Web Services Security: Scenarios and Implementation Guidance for WCF

• Performance Testing Guidance for Web Applications

• Security Engineering Explained

• Team Development with Visual Studio Team Foundation Server

• Windows Azure Security Notes

Developer Guidance Maps

• ADO.NET Developer Guidance Map

• ASP.NET Developer Guidance Map

• Silverlight Developer Guidance Map

• Windows Azure Developer Guidance Map

• Windows Phone Developer Guidance Map

Application Architecture

• A Language for Software Architecture

Books / Guides

• Application Architecture Guide

Methods

• Agile Architecture Method

Guidelines

• Architecture and Design Guidelines

• Mobile Application Archetype

• Rich Client Application Archetype

• Rich Internet Application Archetype

• Services Application Archetype

• Web Application Archetype

• Presentation Layer Guidelines

• Business Layer Guidelines

• Data Access Layer Guidelines

• Service Layer Guidelines

Checklists

• Checklist - Architecture and Design

• Checklist - Communication

• Checklist - Presentation Layer

• Checklist - Business Layer

• Checklist - Data Access Layer

• Checklist - Service Layer

• Checklist - Web Application

• Checklist - Rich Internet Application (RIA)

• Checklist - Rich Client Application

• Checklist - Service Archetype

• Checklist - Mobile Application

How Tos

• How To - Create a Web Application Using Silverlight 2.0

• How To - Create a Windows Application Using WPF and XAML

• How To - Create an N-Tier Application with the .NET Framework

• How To - Design Business Components

• How To - Design Business Entities

• How To - Design Business Workflow Components

• How To - Design Caching for Web Application

• How To - Design Exception Management

• How To - Design Presentation Layer

• How To - Design Rich User Interface with Silverlight and Blend

• How To - Design Your Architecture

• How To - Design Your Data Access Layer

• How To - Domain Driven Design

• How To - Improve User Experience with AJAX

• How To - Page Records Using AJAX

• How To - Structure Your Application

• How To - Use ASP.NET MVC

• How To - Validate Input and Data

Performance

Books / Guides

• Improving .NET Application Performance and Scalability

• Performance Testing Guidance for Web Applications

Methods

• Agile Performance Testing

• Load Testing

• Measuring Performance

• Modeling Application Usage

• Performance Code Inspection

• Performance Design Inspection

• Performance Engineering

• Performance Modeling

• Performance Test Reporting

• Performance Testing

• Stress Testing

• Tuning Performance

Guidelines

• Performance Design Guidelines

• .NET Framework Performance Guidelines

• .NET Interop Performance Guidelines

• ADO.NET Performance Guidelines

• ASP.NET Performance Guidelines

• Remoting Performance Guidelines

• Web Application Performance Design Guidelines

• Web Services Performance Guidelines

• XML Performance Guidelines

Checklists

• Performance Design Checklist

• .NET Framework Performance Checklist

• ADO.NET Performance Checklist

• ASP.NET Performance Checklist

• Enterprise Services Performance Checklist

• Interop Performance Checklist

• Remoting Performance Checklist

• SQL Server Performance Checklist

• Web Services Performance Checklist

• XML Performance Checklist Checklist

Practices at a Glance

• Performance Practices at a Glance

How Tos

• How To - Optimize SQL Indexes

• How To - Optimize SQL Queries

• How To - Page Records in .NET Applications

• How To Performance Capacity Planning for .NET Applications

• How To - Submit and Poll for Long-Running Tasks

• How To - Use Custom Performance Counters from ASP.NET

Security

Guides

• Building Secure ASP.NET Applications

• Improving Web Application Security: Threats and Countermeasures

• Security Engineering

Methods

• Agile Security Engineering

• Security Code Inspection

• Security Deployment Inspection

• Security Design Inspection

• Threat Modeling

Threats and Countermeasures

• Threats and Countermeasures for Web Applications

• Threats and Countermeasures for Web Services

Cheat Sheets

• Web Application Security Frame

• Web Services Security Frame

• Web Services Security Patterns

Guidelines

• .NET Framework Security Guidelines

• ADO.NET Security Guidelines

• ASP.NET Security Guidelines

• WCF Security Guidelines

• Web Application Security Design Guidelines

• Web Services Security Design Guidelines

Checklists

• .NET Framework Security Checklist

• ADO.NET Security Checklist

• ASP.NET Security Checklist

• WCF Security Checklist

Practices at a Glance

• .NET Framework Security Practices at a Glance

• ASP.NET Security Practices at a Glance

• WCF Security Practices at a Glance

Questions and Answers

• ASP.NET Security Questions and Answers

• WCF Security Questions and Answers

Explained

• ASP.NET Forms Authentication Explained

• ASP.NET Windows Authentication Explained

Application Scenarios

• WCF Intranet Scenario - Web to Remote WCF Using Transport Security (Original Caller, TCP)

• WCF Intranet scenario - Web to Remote WCF Using Transport Security (Trusted Subsystem, HTTP)

• WCF Intranet Scenario - Web to Remote WCF Using Transport Security (Trusted Subsystem, TCP)

• WCF Intranet Scenario - Windows Forms to Remote WCF Using Transport Security (Original Caller, TCP)

• WCF Internet Scenario - WCF and ASMX Client to Remote WCF Using Transport Security (Original Caller, HTTP)

• WCF Internet Scenario - Web to Remote WCF Using Transport Security (Trusted Subsystem, TCP)

• WCF Internet Scenario - Windows Forms Client to Remote WCF Using Message Security (Original Caller, HTTP)

• ASP.NET Intranet Scenarios

• ASP.NET Extranet Scenarios

• ASP.NET Internet Scenarios

ASP.NET Security How Tos

• How To - Connect to SQL Server Using SQL Authentication in ASP.NET

• How To - Connect to SQL Server Using Windows Authentication in ASP.NET

• How To - Create a Service Account for ASP.NET Applications

• How To - Improve Security When Hosting Multiple Applications

• How To - Instrument ASP.NET Applications for Security

• How To - Prevent Cross-Site Scripting in ASP.NET

• How To - Protect Forms Authentication in ASP.NET

• How To - Protect from Injection Attacks in ASP.NET

• How To - Project from SQL Injection Attacks in ASP.NET

• How To - Use ADAM for Roles in ASP.NET

• How To - Use Authorization Manager (AzMan) in ASP.NET

• How To - Use Forms Authentication with Active Directory in ASP.NET

• How To - Use Forms Authentication with Active Directory in Multiple Domains

• How To - Use Forms Authentication with SQL Server in ASP.NET

• How To - Use Health Monitoring in ASP.NET

• How To - Use Impersonation and Delegation in ASP.NET

• How To - Use Protocol Transition and Constrained Delegation in ASP.NET

• How To - Use Regular Expressions to Constrain Input in ASP.NET

• How To - Use the Network Services Account to Access Resources in ASP.NET

• How To - Use Role Manager in ASP.NET

• How To - Use Windows Authentication in ASP.NET 2.0

WCF Security How Tos

• How to: Audit and Log Security Events in WCF Calling from Windows Forms

• How to: Create and Install Temporary Certificates in WCF for Message Security During Development

• How to: Create and Install Temporary Certificates in WCF for Transport Security During      
  Development

• How to: Create and Install Temporary Client Certificates in WCF During Development

• How to: Host WCF in a Windows Service Using TCP

• How to: Impersonate the Original Caller in WCF Calling from a Web Application

• How to: Impersonate the Original Caller in WCF Calling from Windows Forms

• How to: Perform Input Validation in WCF

• How to: Perform Message Validation with Schema Validation in WCF

• How to: Use basicHttpBinding with Windows Authentication and TransportCredentialOnly in WCF from Windows Forms

• How to: Use Certificate Authentication and Message Security in WCF Calling from Windows Forms

• How to: Use Certificate Authentication and Transport Security in WCF Calling from Windows Forms

• How to: Use Delegation for Flowing the Original Caller Credentials to the Back End in WCF Calling from Windows Forms

• How to: Use Health Monitoring to Instrument a WCF Service for Security

• How to: Use netTcpBinding with Windows Authentication and Message Security in WCF from Windows Forms

• How to: Use netTcpBinding with Windows Authentication and Transport Security in WCF from Windows Forms

• How to: Use Protocol Transition for Impersonating and Delegating the Original Caller in WCF

• How to: Use the SQL Server Role Provider with Username Authentication in WCF Calling from Windows Forms

• How to: Use the SQL Server Role Provider with Windows Authentication in WCF Calling from Windows Forms

• How to: Use Username Authentication with the SQL Server Membership Provider and Message Security in WCF Calling from Windows Forms

• How to: Use Username Authentication with Transport Security in WCF Calling from Windows Forms

• How to: Use wsHttpBinding with Username Authentication and TransportWithMessageCredential in WCF Calling from Windows Forms

• How to: Use wsHttpBinding with Windows Authentication and Message Security in WCF Calling from Windows Forms

• How to: Use wsHttpBinding with Windows Authentication and Transport Security in WCF Calling from Windows Forms

Visual Studio Team System

Guides

• Team Development with Visual Studio Team Foundation Server

Guidelines

• Team Build Guidelines

• Source Control Guidelines

• Reporting Guidelines

• Project Management Guidelines

Practices at a Glance

• Team Build Practices at a Glance

• Source Control Practices at a Glance

• Reporting Practices at a Glance

• Project Management Practices at a Glance

Questions and Answers

• Source Control Practices at a Glance

How Tos

• How To: Add a New Developer to Your Project in Visual Studio 2005 Team Foundation Server

• How To: Automatically Run Code Analysis with Team Build in Visual Studio Team Foundation Server

• How To: Create a Custom Report for Visual Studio Team Foundation Server

• How To: Create a “Risk over Time” Report for Visual Studio Team Foundation Server

• How To: Create Custom Check-in Policies in Visual Studio Team Foundation Server

• How To: Create Your Source Tree in Visual Studio Team Foundation Server

• How To: Customize a Process Template in Visual Studio Team Foundation Server

• How To: Customize a Report in Visual Studio Team Foundation Server

• How To: Manage Projects in Visual Studio Team Foundation Server

• How To: Migrate Source Code to Team Foundation Server from Visual Source Safe

• How To: Perform a Baseless Merge in Visual Studio Team Foundation Server

• How To: Set Up a Continuous Integration Build in Visual Studio Team Foundation Server

• How To: Set Up a Scheduled Build in Visual Studio Team Foundation Server

• How To: Structure ASP.NET Applications in Visual Studio Team Foundation Server

• How To: Structure Windows Applications in Visual Studio Team Foundation Server

• How To: Structure Your Source Control Folders in Team Foundation Server

My Related Posts

• 10 Years at patterns & practices

• People I Worked with On Past Projects

• The Power of Blue Books for Platform Impact