The challenges of a software API Design

Designing APIs poses multiple challenges, among them:

• APIs are targeted at developers
• APIs have 'stickiness'. A bad APIs is a nightmare for years as many internal and extenal apps depend on it.
• APIs are shared by many applications, so problems can impact multiple applications (differently)
• APIs must be discoverable, intuitive, and use consistent conventions
• Versioning and backwards compatibility on change
• Good (and updated/correct) documentation is required
• Likewise, automated testing is required

But there are benefits for exposing APIs even internally:

• hide implementation
• reuse code
• reduce duplication
• easier to optimize

Exposing APIs externally has many market benefits that we'll discuss separately. Just note that all major platforms (Facebook, Linkedin, Windows/Win32) are really APIs; and being a 'platform' can be very beneficial to a provider.

Good Writing using the Madman Architect Carpenter Judge system

Write in four phases

Madman 
- jot down all ideas
Architect
- look at your ideas. try to think of a sensible order. identify three main points
- now you have an outline
Carpenter
- rapidly, write down paragraphs in support of the outline
- don't edit in this phase
Judge
- Now review everything you wrote, and try to judge it
- How would someone who is not friendly to me look at it.
- Would it look self serving? insincere? am I making unjustified claims? etc..


HT to guide to better business writing

Optimize Your Life with Lean Software Development

Lean (a concept in manufacturing optimization, and now in software development) has some general 'life-lessons', focusing on "Add Nothing But Value - eliminate waste".

* Eliminate Overproduction
In software, unneeded features.
In life - eliminate hoarding, interruptions, unneeded focus? or is this stretching the analogy too far?

Innovation Technique: Job Scoping

For a specific Job To Be Done, try to answer:
1. What is a broader problem, and why it is important
2. What is a narrower problem, and what is the barrier that makes this narrower problem important to the JTBD

Do those answers provide any useful insights?

Internal vs. External Quality

Internal vs external quality OR why software design matters.
 Internal quality (known bugs, code quality, design, extensibility, refactoring) doesn't matter to the customer NOW. The customer may not be able to conduct any (blockbox) tests to determine the internal quality of the software anyhow, so it may as well not exist.
So... marketing/management may be tempted to concur.
They would be wrong... because, in time code quality impacts the bug rates, performance, cost of development of new features, and time to market for new features. The current code base results in debt. Like all debt, you can pay off the principal early or pay off the interest on an ongoing base. Either decision is fine in some circumstances, as long as it is the right decision for the organization
==> more important to keep the changing and critical areas 'clean'.

Sources of tech debt
=> some debt is taken by design
==> some by being reckless
===> and some debt is only visible in retrospective as we learn more about the problem space.

Predicting the next new song

A few interesting questions in the cross of 'big data' and music:
1/ Predicting the next song a -particular- user will like. (e.g. Pandora)
2/ Predicting the next song that will become popular
 Last I heard, Pandora uses analysis of the music itself. But there is so much data out there:
* The singer
* The Lyrics
* Social networking and google mentions
* Radio plays
* The label (is a valid datapoint)

Innovation technique: Nine Windows

consider the matrix {past,present,future}x{supersystem,system,subsystem}. Fill in center(present,system). Complete the grid. Observe for insights.

Innovation Technique: Outcome Expectation

1. Identify Jobs To Be Done 2. Identify 4 quadrants: desired/undesired outcomes to client/provider 3. Create outcome statement: include the direction of action (minimize/maximize), unit (time/cost/etc.) object, and context. e.g.: "Decrease the likelihood of delayed customer orders" Software Industry Relevance: Often there are 3 players: the end user (functionality), the provider as a financial entity (monitezation) and the provider as a technical entity (design integrity). You need to recognize those disparate needs and identify what each role/entity preferences are. Based in part on The Innovator Toolkit

Innovation Technique: JTBD (Jobs To Be Done)

Jobs To Be Done: Instead of focusing on what you are doing, focus on the jobs your customers (internal or external) are trying to accomplish. Instead of improving the lawnmowers you are manufacturing, maybe you should genetically engineer grass? (the job to be done: keep lawn looking tidy). Software industry relevance: similar to User Stories, but much more higher level than most people use User Stories. Based in part on The Innovators Toolkit

Personal website redesigned

My personal website underwent some modernization. Let me know what you'll think!

Twitter - at last.

For short and succinct words of wisdom... use a dictionary. For everything else, there's @yanivpessach

Distributed Storage eBook available

Good News Everyone! My 'Distributed Storage: Concepts, Algorithms, and Implementation' eBook is now available on Amazon. The eBook is a short introduction to the topic and is targeted at the academic level as an introduction to the topic.

Agile, Scrum, And PMI project management

As both a Scrum Master and a PMP-certified project manager, the conflict between these two approaches is something I have to deal with on a daily basis.

I noticed the guys in SprintPlanning.com are offering a free agile and planning online course about it. Worth checking out.

The most important part of Scrum

There is an ongoing debate in the agile/scrum community about 'what aspect of scrum is most important'.
Obviously, the most *recognizable* aspect of scrum is the daily stand-up meeting. But - is this meeting so important? I've seen and ran Scrum teams that skipped the daily standup, had a daily standup meeting twice a week, or otherwise modified the ritual. And they worked well.
In my opinion, the two most important concepts in scrum (which are general Agile concepts), are *timeBoxing* and *adaptive iteration*. This is why I think the Sprint Planning meeting is the most crucial - this is when planning ofr the next iteration, taking into account changes that need to be made, takes place.
It is true that much of the 'lessons learned' are acquired in the *sprint retrospective* meeting, if it is held. But without incorporating this information in the next sprint planning session, the retrospective degrades to nothing but a 'whine session.

The scientific method to Unit Testing

For many, Unit Testing is how we prove to ourselves (and others) that our code works, at least initially. And yet, many unit tests fail to find the very reproducible, localized bugs they were to prevent. One reason this happens is our natural tendency to create unit tests to prove that our code works, rather than to prove that it fails.
The scientific method suggests the opposite direction. A claim is scientific if it is falsifiable .
A good unit test (and a good acceptance test) will try to prove the code wrong, by including as many edge cases as possible, and by challenging the code invariants.
Good testers know this, and spend much time looking for edge cases. But we developers should keep the same mind-frame, and try to prove our own code wrong.
It's OK.
If the code is proven wrong by your unit tests, you just fix your code. You didn't check in code before writing your unit tests, now did you? :)

Disabling right click and other programming insights

Did you ever wander into a website that disables your right click (possibly, as a way to block you from using 'save image as') and wonder how to disable that obnoxious behavior?

Well, a simple view source shows how the right click behavior was taken over. They set document.oncontextmenu (or onclick, or mousedown) to their own method, returning 'false'.
something like: document.oncontextmenu = function("alert('noooo');return false");

A simple script (which can easily be used as a bookmarklet) can counter this behavior:

javascript:var x = (document.onmousedown==null?'':'mousedown ') + (document.onclick==null?'':'click ') + (document.oncontextmenu==null?'':'contextmenu ');if (x=='') x='none';alert('Yaniv says: detected '+x);void(document.onmousedown=null);void(document.onclick=null);void(document.oncontextmenu=null)


How would I write a website where the above script does not work?
Well, one approach would be to repeatedly set document.oncontextmenu to the 'disable' method, using a timer.
[To Be continued]

Execute any Windows API from the commandline with RunAnyDll

The RunAnyDll tool can be used to call any Windows API from the command line.

RunAnyDll lets you execute any Windows API from the command line, batch files, startup menu.. anywhere...
And, unlike RunDll32, the RunAnyDll tool is not limited to APIs with specific .

Call MessageBox to get a message box, GetSystemPowerStatus to get available battery power, GetConsoleTitle for the current cmd title, and 1000s more APIs.

Example Usage:
To open a messageBox by calling the MessageBox API:
RunAnyDLL user32.dll MessageBoxA UINT 0 LPSTR Welcome LPSTR goodbye UINT 1

To get the current system power status:
RunAnyDll kernel32.dll GetSystemPowerStatus LPBYTE 00

To emit an annoying beep through the computer speakers
RunAnyDLL Kernel32.dll Beep UINT 1200 UINT 1000

Background:
RunDll32.exe, part of Windows, can used to run an entry point (function) in a dll. As such, it can be used for many amazing purposes, without writing new code.
However, RunDll32 expects the API called to have 4 specific params: HWND, HINSTANCE, LPSTR, and int. While it still can be used for calling some APIs accepting fewer params, this is not reliable/recommended.
(Discussion at: http://blogs.msdn.com/oldnewthing/archive/2007/06/07/3128210.aspx)


More information and download here

Lean Software Development according to Poppendieck

To create a Lead process, waste should be eliminated.

Taking this approach to software development (from requirements to deployments) helps explain many of the project failures I have saw or heard about.


In software development, the following are considered waste:

1. Partially Done Work
This is hard to first see, but a module that 'almost' works (but then abandoned for months) has very little value. So is a module that is so-called complete, but is untested, and therefore the bug-fixing work is not done.
This is similar to 'inventory' in classic lean; there is a high cost in having partially-done software (the knowledge gets obsolete, APIs and other systems shift, etc) which is not obvious to management.
Also, if it isn't deployed now, there is a chance that it would never be deployed.

2. Extra process
Clearly, stacks of paper have no positive impact except whatever benefit they did to the shipping code.

3. Extra features
Any additional line of code, and every additional feature, adds system complexity. Fixing a bug would now cost more; adding other (important) features would take longer. Carrying 500 pounds of dead weight in your car is only going to slow you down, and so are deadweight features

4. Task Switching
Every time a developer has to perform multiple tasks, they all take longer, and focus is lost.

5. Waiting
Having to wait (for approval, feedback, etc) slows the project down and wastes time

6. Motion
Especially in document handoffs, when every time a document is passed to the next person, knowledge is lost.

7. Defects
Bugs slow the process down since they require fixing, testing, and can cause other types of slowdown (such as causing developers to lose focus on their new features). Buggy code takes longer to complete.

Lean Software Development

Wikipedia states that "Lean Software Development is a translation of lean manufacturing principles and practices to the software development domain."

Lean software development, to me, is best viewed as both the philosophical underpinning of Agile, and as the extension of a similar approach to the entire software 'production' chain, from conception, through approval, to deployment.

The main points that a developer or a program manager would find helpful are:
* System: Optimize the system, not each part separately
* Value: understand what creates value for your customer, and how your organization creates value
* Flow: Maximizing speed minimizes wastes
This one is hard to see at first. But delivering a functioning solution earlier is critical since:
- You are producing the solution since it has value to the customer. The sooner the customer uses a functioning system, the more time he gets the benefit for, so the higher his total benefit is. If the core systems would save the customer $100k a month, delivering a core system 3 months later just cost $300k
- The time it takes to initiate and approve a project counts as cost, too. That stack of 'waiting for approval' projects on the VP's desk is costing the organization, the same way that the queue of people waiting for service in the grocery store costs you time.
- The more 'outstanding' projects there are, the more multitasking people need to do. Multitasking is expensive and inefficient.
- The more time passes, the less knowledge is fresh. Knowledge is what software development is all about.
* Pull: Deliver value based on customer (or next-in-production-chain) needs
And the perfect bad example is when the test organization is months behind the developers in testing. The dev team might be code-complete, but the product won't be shipping for six more months...


Special thanks to Alan Shalloway and his Lean presentation. The errors, of course, are all mine.

12 Basic Principles of Design For Perfomance

Goals and measurement:

1. Set performance goals
2. Set a performance 'budget' (e.g. 300ms for update etc)
3. Define performance testcases

Design:

4. Understand resources - disk, memory, network bandwidth, database connections, and local data protected by locks
5. Cut the deadwood - especially when reusing existing services and components, make sure your code doesn't do more (create more results, processes more data) than is needed.
6. Batch work - this will minimize roundtrips and enable efficiency gains from accessing resources only once.
7. Pool shared resources - such as database connections, TCP connections, or anything which has setup and teardown costs
8. Process independent tasks concurrently
9. Consider affinity - put resources near where they will be used; duplicate readonly resources (trade-off storage/memory for access time)
10. Use caching; design appropriate cache policy
11. Consider the efficiency of your algorithms; prototype algorithms early
12. Identify bottlenecks; there is always one more bottleneck