Showing that fear of mediocrity can make the world a better place.


Distributed integrated developers for progressive agile in the enterprise

Agile is a dangerous word today. It has been picked up by the buzz-word industry and the spin is interesting to watch. The interesting thing is that I’m not sure that the current agile practices have finished their transformation. We seem to be freezing the progress at a comfortable point and pumping out scrum masters. I’d like to think that the future of agile lies in a more complete integration and alignment of developers and the business they support. Allow me to present my humble vision for distributed integrated developers.

Scattered to the winds

First, a picture.

Distributed integrated developers

Now a thousand words.

(Warning: N=1 study ahead) My experience has pointed me in a general direction. All the technical prowess in the world does not make up for solving the wrong problem. In most cases the wrong problem is solved due to miscommunication and process built up around a relatively simple task. If people had taken the time to question what they were hearing and make sure everyone was in agreement then they would have saved countless hours of pain. It seems that the further the developer is from the action of the business users the worse the miscommunication and the project problems. I’ve had issues clear up immediately by just going and watching business users perform the actions that they normally would. The clarity and insight that comes from direct observation cannot be understated. The agile manifesto clearly states “customer collaboration over contract negotiation” so where did we go wrong?

Get uncomfortably close to your users

Enterprise developers are usually extremely lucky in that they have immediate access to their software user’s domain. They can ask them questions, do usability studies, and quickly iterate in a relatively low risk environment. Unfortunately that rarely happens. For one reason or another the development department is walled off from the business and in this isolated environment loses the benefit of being an in-house development department. My approach, as shown in the diagram, would be to plant a development group (ba, pm, qa, devs) directly into the department that they are supporting. The ability to work alongside and react to changing needs would greatly improve the groups agility. The development department then becomes purely the architecture group who are governing across and spreading the best practices between each distributed integrated development group. The architects could move in and out of the development group to work alongside the business and make sure their designs are effective in the trenches.

The theory is nice, but in practice it would <insert pessimistic view here>

The business unit would need to have visibility into the distributed integrated development group’s work. Standard agile practices should suffice here. The ability to clearly communicate priorities would get better over time as the business group gets comfortable with how things get developed and how reactive the team is to help pitch in during emergencies. The distributed integrated development group would need to possess discipline so the business unit did not drive and overrun them since they are outnumbered. This is where the architects’ guidance and firm hand would be helpful.

In summary, developers need to stop putting themselves on a pedestal as “blessed designers”. Developers are no better or worse than other members of the business. The development department should plant developers directly into the business. The agile revolution has not finished and cementing the process and collaboration at this point might be premature. Tear down the wall!

Feel free to try this and let me know how it works for you.

The details don’t matter until they do

Dualities rarely exist in reality. There is no yes and no, on and off, true and false. Almost everything in the world exists in part of a continuum; a sliding scale between two extremes. There is absolute zero and possibly an absolute hot (absolute hot is still up for debate) but there is also an entire range in between. If you can’t see the shades of gray in the topic you are working on then you haven’t looked close enough. But let’s suspend all that continuum talk and focus on a simple duality: abstract and concrete.

Rising levels of abstraction

The most concrete part of computers are the circuitry and the 1s and 0s that make it all work. At the end of the day everything is a 1 or a 0 and the rules of physics and electrical current apply. The further away from that you get the more abstract things can be. When you get high in the ether that is enterprise systems everything has a tendency to become very abstract. The concepts are more about thoughts and ideas made to represent a made up reality. You are building a fantasy land with made up rules (although people take those rules very seriously). And for a while the concrete doesn’t matter. How is data written to disk? Doesn’t matter. We have an abstract idea of a datastore with a save method. Done. And it works. Until it doesn’t. Then the concrete details matter and you damn well better be able to chase the problem down the stack until you can reason about the problem.

It’s an implementation detail

In many cases the implementation details, the concrete implementation, doesn’t matter. Certainly in most business applications they don’t care how it is implemented only that it is and it works (I would say reliably but that is up for debate). When you are designing a large system you start to get an idea what the main pieces are and you can comfortably delay choosing an implementation until you absolutely have to make a choice. You can delay the implementation and the details if you don’t need to be concerned with high performance and mission critical reliability. I’m not talking the “we need the report now” type of mission critical. I’m thinking more along the lines of “rocket two isn’t responding. prepare for crash landing” kind of critical. When you start trying to eek out the most performance possible or uptime and reliability has to be greater than six nines then the concrete matters. Your abstractions break down and how the 1s and 0s move across the physical circuit makes a difference.

It’s a numbers game/Know your abstraction

I personally like programming at a high level of abstraction. The further from the concrete the better. Even when painting I prefer abstract and surrealism to realism. But that’s not for everyone. There is absolutely a place for concrete programming just as there is a demand for realist paintings. The problem I have is that most programmers are too focused on the concrete and too focused on the implementation. Maybe it’s technology infatuation or just a desire to work in a certain level of abstraction. Either way I would argue that 80% of modern business application development is in highly abstracted areas.

In the end you will be happier when you know what level of abstraction is appropriate for the current problem. If you insist on always talking about MRUs and LRUs then don’t be surprised when a business user rolls their eyes and doesn’t trust you. On the other end of the spectrum if are talking in the language the business understands but you system doesn’t perform well then you had better be able to step away from Visio long enough to determine where in your amazing architecture you introduced a bottleneck.

In most cases it’s amazing how little the details of the concrete matter in programming. Unfortunately they won’t matter until it’s critical so ignore them at your own risk.

Sow needs. Followup with solution.

Marketing and engineering are two sides of a simple duality. Engineering is the solution or product that solves a need. Marketing is sowing the need into the marketplace. An engineer can design a shovel but it is up to marketing to convince people that the shovel solves the problem that is currently caused by digging with sticks.

Filling the need by patchwork

The typical problem space for just about everything in life looks something like this:

The dips represent gaps in the workflow where things aren’t as optimal as they could be. These gaps are needs caused by the incongruously patched together workflow. In essence pieces of the problem space leak though causing suboptimal experiences and frustration. They become annoying potholes in the road to whatever it is we are trying to accomplish. Most products and marketing aim to make people aware of these potholes and how the product fills one or more of them. The holes are patched one by one by different businesses and industries with varying levels of success.

Most businesses operate in this patchwork manner. They have a vested interest in continuing to patch things because it doesn’t disrupt their previous patches which may be a source of income. Most businesses also aren’t concerned with the customer and the actual workflow the customer goes through. Whether it is because they are too close to the domain in question or they are too near-sighted most businesses cannot create realistic use cases that envision the whole process.

Removing the need by repaving

Every now and then the problem space gets re-imagined. Someone takes a step back and questions a basic assumption that when answered can push the entire problem space forward. Whether by dumb luck or visionary insight they strip the context to its core and shape the optimal solution around it.

Now the problem looks like this:

They essentially re-imagine the workflow and by stripping the problem down to it’s base can repave and rebuild the solution without trying to just fit it into the gaps. This is rare and usually changes the face of an industry or problem space. Most of the time this kind of work is done by people who have no vested interest in the current solution to the gaps and are freer to innovate them into oblivion. The common term is startup or entrepreneur but anyone with an idea and the ability to implement it can do this. Unfortunately good ideas well executed are difficult to come by.

Incremental or leap in progress you still need to sow the need

It seems like the more of a leap in progress the idea is the more that marketing needs to sow the need into the marketplace. Like planting a seed knowing that it will take time to catch root and grow you need to put the idea out into the market and make people question the current solutions and why they don’t address the fundamental nature of the problem. By the time they are begging for a solution you can bring your product to market and reap the mind-share that you have sown. The difficult part is really owning the customer’s workflow across any arbitrary industry and system boundary and providing the seamless experience.

Edit: For a concrete example of this idea see

Polyglot programming explained

Learning multiple programming languages is similar to learning multiple religions. Each one you learn decreases your zealotry for any one in particular.

It’s all in your head

Pragmatic developers are constantly making trade-offs in the design of the code. Sometime they take on technical debt because it will be easier to pay it off later when things are clearer. The problem is what happens when they leave before they get a chance to pay off the debt and clear the backlog of ideas out of their head.

Documenting design

I know. Documentation is a four letter word for developers. But it doesn’t have to be. Yes the code should be self documenting but how do you document what the code could have been? How do you document the trade-offs and micro-decisions that led to the current code? In most cases this is left up to the maintaining programmer as they go spelunking in the depths of your code tryng to use software archaeology to unearth what the heck you were thinking when you wrote the code. In some systems only the penitent man shall pass. This Dr. Dobbs article on documenting micro-decisions provides a really simple idea. Track the decisions you make next to or in the code and make it part of your process. Then at least when you go back and re-read your code wondering what you were thinking there will be some context around the thought process you were going through.

Pair programming

Another effective practice is pair programming. In this case the knowledge is shared by at least two programmers and they can back each other up in the event the other is unavailable or cannot recall why they made the decision they did.

Capture it for yourself

If you use the documentation then you will see what makes it effective and what is a waste of time. Whenever you run into a question of why something is the way it is check your decision document. If it doesn’t have the answer then add it. It’s not a silver bullet and it’s not the sexy side of development but it’s simple enough to try out and determine if it’s worth adding it to your tool belt. Like TDD it seems like it would really pay off in frantic and chaotic environments. If you are a team leader then try asking your team to document the design decisions for a week and ask review them at the end of the week. It might also be a way to catch design mistakes and incorrect assumptions early in the process.

Edit: The problem with continuous learning is nothing is ever finished. No sooner do I post this then I finally get around to watching by Dan North. It’s a great talk which confirms my opinion that if you can’t weigh the trade-offs of a decision then you are just guessing at things. Very few people seem to know why they do things the way they do. The rest seem to be fine cargo-culting ( their way through a job.

One process, two process, red tape, duct tape

Not invented here (NIH) in one part

AKA “Sorry future maintainers”

Peasant (to court)

Good morning. I am a lowly office peasant from the far north wing. I humbly submit my request to the benevolent kings and queens of finance. I have through much personal and professional hardship arrived at a reasonable solution to the problem with which you have tasked me. Your frustration with the current state has led me down many roads and journeys in search of new tools and technology that will help me fix what pains you. Through my own trials and tribulations I have arrived at a well thought and proven solution. All I request of the merciful court is a small allowance in order to purchase and setup the solution so that we may get back to giving praise to you, our kings and queens. This small pittance of money is nothing compared to the gains we will make using the new technology and your honors will surely be praised throughout the land for handily solving such a notorious problem.

The Court

Proposal REJECTED by order of the kings and queens. The court recommends you figure out how to solve the problem they presented without the needed tools.

Peasant (to court)

Honorable kings and queens of the court. I humbly request you to reconsider your contribution to solving the problem you requested of me. Without the proper tools that are already available with a small allowance from your generous purse we may not be able to adequately solve the requested problem. The solution is complete and ready to be implemented with your blessing and minimal effort. Any other path will cost ten times as much and will not be of your majesty’s high standards. For the honor and future safety of you kingdom please allow my humble request to be answered.

The Court

Proposal REJECTED.

Peasant (to other peasants)

My brothers and sisters, our pleas have fallen on deaf ears and our prayers unanswered. The tools and technology that we need to work for our livelihood have been rejected. We have requested a plow and been told to use a stick. We have requested a mule in triplicate and have been told to use our mangy dogs. We need to continue to work and produce goods or we shall surely starve but the tools that help us to produce more have been dismissed out of hand by those who know not how to use the tools.

Future generations will look at this turning point and question the futility of our decision and many generations will suffer with inferior tools. Those that come after us will curse our name instead of the true tyrants but we must stand strong. This decision was not an easy one and it has weighed heavily on my soul.

I will not stand idly by and watch my brethren use sticks to plow and dogs in place of mules. I have traveled to many locations and seen how the plow and mule are created and I believe we can make them for ourselves. Sure it will be expensive. We will lose much time building what has already been created elsewhere. We will lose time and money as the home made tools break down and don’t keep up with the latest tools but it will be better than nothing. I do not want to make these tools when we could be better spending this time growing amazing new products that will bring much praise to our small kingdom. We will not let the tyranny of the court decide the quality of our lives and we will become masters of our own destiny. I do not want to start down this road but my hand has been forced. We will enjoy the prosperity of the enlightened and we will bring it by our own hand. If the courts will not provide then we shall supply ourselves with the sweat from our brow.

We shall create.

The last shall be first…in F#

Just pathetic

One of the first things I used F# for outside of demo learning apps was some scientific data analysis. One of the desired results was looking for the first and the last occurrence in a set of data. My first attempt at this is pretty pathetic. Feel free to point and laugh at this late night approach.

I was trying to figure out how to get access to a specific element in a sequence. Since I wasn’t too familiar with the F# libraries I came up with a horrible but usable solution. This code didn’t need to be fast, which is good since there was an obvious performance issue. Using my limited knowledge of accessing data in a sequence I used Seq.head to take the first element. This solved half the problem. Now I just needed to get the last element in the sequence. For some reason I came up with converting the sequence to a list and reversing the list. Then I could take the head (first element) again and that one would be the last element. It was convoluted and I knew it but it worked and I was tired.


Then I recently re-watched and something jumped out at me. Seq.nth. That little three letter bastard of a function. One thing I noticed about functional programming is that the discovery of methods/functions is a little different than in OOP. Instead of asking an object what it can do you have to interrogate some library and see if it can accept what you want to pass it. Enough excuses as to why I missed it. I was able to quickly encapsulate what I wanted with the following code.

I like this code much better. It feels much less like an ugly hack. It also follows more in line with functional programming where you specify what you want not how to do it. See for more information on functional thinking.

Thank god for refactoring.