I’ve been experimenting with matplotlib recently, both interactively in an ipython shell as well as non-interactively as a chart image generator to be served through the web.

This post shares some tips that took some searching on how matplotlib operates in different interaction contexts.

General Concept: Backends

matplotlib has the concept of backends — essentially the target canvas or surface it can render a figure to.

A backend implementation takes matplotlib’s internal representations of high-level drawing objects such as lines and axes and converts these to a form suitable for that backend.

For example, an image backend — effectively what we use when we save a figure to a file or stream — will know about different image formats, and how to convert drawing objects into pixels in that format.

The key point here is that interactive and non-interactive modes require a backend implementation suitable specifically to each of them.

The following ipython sample session illustrates common backend API calls which we will need in later sections:

In [1]: import matplotlib as mpl                                                 

In [2]: mpl.get_backend()                                                        

Out[2]: 'Qt5Agg'

In [3]:  


In [3]: %matplotlib --list                                                       

Available matplotlib backends: ['tk', 'gtk', 'gtk3', 'wx', 'qt4', 'qt5', 'qt', 
'osx', 'nbagg', 'notebook', 'agg', 'svg', 'pdf', 'ps', 'inline', 'ipympl', 
'widget']


In [7]: mpl.use('pdf')


In [8]: mpl.get_backend()

Out[8]: 'pdf'

Note that simply doing mpl.use('qt5') fails. The suffix agg needs to be added in some cases — but obviously not all, as per the pdf example above (go figure) — like so:

mpl.use('qt5agg')

It is not case-sensitive, so Qt5Agg and qt5agg work just the same. The exception that shows actually lists the proper names:

ValueError: Unrecognized backend string 'gtk': valid strings are ['GTK3Agg', 
'GTK3Cairo', 'MacOSX', 'nbAgg', 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo', 
'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo', 'agg', 'cairo', 
'pdf', 'pgf', 'ps', 'svg', 'template']  

Furthermore, even though ipython lists those various backends, it may still fail on calling use(...), and require you to install additional components.

Basically, ipython is aware of a bunch of possible backends, and attempts to dynamically load the one requested by matplotlib.use(...).

Switching backends can happen prior to any plotting command, e.g. matplotlib.pyplot.plot(..).

In [61]: mpl.use('agg')                                                          

In [62]: plt.bar(x, y)                                                           

Out[62]: <BarContainer object of 25 artists>

In [63]: plt.show()                                                              

/home/xxx/apps/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py:445
: UserWarning: Matplotlib is currently using agg, which is a non-GUI 
backend, so cannot show the figure.
  % get_backend())

In [64]: mpl.use('qt5agg')                                                       

In [66]: plt.bar(x, y)                                                           

Out[66]: <BarContainer object of 25 artists>

In [67]: plt.show()                                                              

Interactive Mode

In an interactive shell such as ipython, we want an interactive backend such as Qt5Agg or TkAgg.

TkAgg in particular, was what worked for me on Mac with an Anaconda distribution without installing anything further.

These are image renderer and viewer implementations with controls for interactive use.

When exploring in interactive mode, I was confused why matplotlib.pyplot.show() was only taking effect once: calling show() then closing the chart window, and calling show() again does nothing.

Essentially, one needs to reissue a plotting command such as pyplot.bar(...) or pyplot.plot(...) to force a draw, and then call show() again.

Of course, simply starting afresh with a new figure also works, since a new figure is a fresh draw.

plt.figure()
plt.show()

plt.figure()
plt.show()

Non-Interactive Mode

A common use case for non-interactive mode is to produce and serve graphics over the web. The server code will run completely headless and not require any GUI toolkits to be installed where it is deployed.

We will often get user inputs, plot the figure in-memory, then stream it through, encoded as an image to be rendered on the browser, or saved to disk, etc.

The correct backend to use for this is agg.

Essentially, as part of bootstrapping our application, webapp or otherwise, we need to run the following prior to any plotting API calls being made:

import matplotlib
matplotlib.use('Agg') # case-insensitive

In most frameworks, such as Flask, this can be done in the web module’s __init__.py where we create the application object.

Of course, this configuration appropriately stops show() from working, issuing a UserWarning:

In [61]: mpl.use('agg')                                                      
                                        

In [62]: plt.bar(x, y)                                                       
                                        
Out[62]: <BarContainer object of 25 artists>

In [63]: plt.show()                                                          
                                        
/home/xxx/apps/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py:445
: UserWarning: Matplotlib is currently using agg, which is a non-GUI 
backend, so cannot show the figure.
  % get_backend())

Happy Hacking!

CSSCDA+JQ™ – an acronym that is nearly as bad as what you think it means.

I’ll keep this rant short, and mention no names to protect both guilty and innocent. I even deferred writing this so my trauma could heal in the interim.

If you wish, skip over to the “why” of this rant.

Anyway, here’s the essence of CSSCDA+JQ™ “pattern”.

The “Pattern”

Rationale

Add dynamic behaviour to a website, that is actually a web application, but refuses to acknowledge this identity. Essentially, manage the site’s identity crisis.

Approach

First, we have many — reasonably modularised (no sarcasm) — HTML templates like so:

 1<!-- template foo -->  
 2<div class="chupacabra-button">
 3  ...
 4</div>
 5
 6<!-- template bar -->  
 7<div class="spaghetti-monster-magic">
 8  ...
 9</div>
10
11<!-- template baz -->  
12<div class="yeti-lord-magic">
13  ...
14</div>
15
16<!-- template meow -->  
17<div class="pastafarian-magic">
18  ...
19</div>

Next we have a top-level site template, with 30 million conditional statements (I exaggerate, but it sure feels that way, don’t invalidate my feelings, yo):

 1<html>
 2 ...
 3 <!-- if $foo -->
 4 #include bar.js
 5 #include baz.js
 6 <!-- fi -->
 7 <!-- if $baz -->
 8 #include foo.js
 9 #include baz.js
10 <!-- fi -->
11 <!-- if $bar -->
12 #include meow.js
13 #include baz.js
14 <!-- fi -->
15 <!-- if $meow -->
16 #include woof.js
17 #include kapow.js
18 <!-- fi -->
19 <!-- if $dingding -->
20 #include urgh.js
21 #include baz.js
22 <!-- fi -->
23 <!-- if $woof -->
24 #include lala.js
25 <!-- fi -->
26 ... to 30 million 
27 ...
28</html>

Then, we have individual JS files referenced above, using plenty of JQuery to attach behaviour, like so:

1// woof.js
2$('.pastafarian-magic').on('...', function magic() {...})
3
4// ditto: (bar|baz|foo|kapow|urgh|lala..30million).js
5$('.yeti-lord-magic-ohgno-typo-in-css-class').on('...', function magic() {...});
6

Finally, you have — reasonably well-modularised (again, no sarcasm here) — controllers, setting the conditional variables referenced in the site template, and including in their relevant specific templates.

1class PastaController {
2  def foo() {
3    // ...
4    ctx.set('dingding', true);
5    ctx.template('woof');
6    // ...
7  }
8}

Ah, this marvellous application of Stringly-Typed Programming when so much behaviour is hooked in via includes, CSS classes, and many, many snippets of all-over-the-place jQuery (some containing direct DOM mark-up in — you guessed it — JS strings).

Complexity

So our potential Big-O complexity, most rigorously calculated by yours truly, is:

1O(
2  Controller Count * 
3  Template Count * 
4  JS Files Count * 
5  Magic CSS Classes Count
6)
7

In other words, we have reached O(HGNO) + O(MG) complexity. I’d say very, very close to O(KMN) complexity.

At this point, when project leads still think that bringing in a framework to manage the complexity is too much overhead, and they like that the CSSCDA+JQ™ approach is lightweight, well… it is best to find another project and/or project lead.

Simplicity, Complexity, and Cognitive Weight

I’m all for using the right tool for the job. I have biases towards those tools, but my rant above is based on very real problems: brittle, hard to follow code, with every small change taking much more time than it ideally should. And let’s not forget refactoring (and thus testing) concerns.

It’s important to understand complexity and cognitive weight:

A method with a long switch statement, but where each branch does very simple, exact tasks, e.g. a very common pattern used in simple parsers, may be long, and look ugly, but inherently, it is flat, and does not impose an excess cognitive load on the reader. Ignore for the moment using polymorphism or similar approaches for scaling better.

With CSSCDA+JQ™, on the other hand:

• Yes, JQuery is simple
• Yes, inherently attaching behaviour to CSS-marked elements is a common, simple pattern (yes, simple until you consider scoping, take that this).

However, it has a reached a point where, taken as a whole, the individual, simpler ideas, no longer show that simplicity. The sum of the parts is far more complex than the parts.

At this point, a newer abstraction like an MVC framework ala Vue or Angular, one that may indeed be — by itself — more complex than the individual parts that make up CSSCDA+JQ™ becomes justified because it unifies the overall concepts that CSSCDA+JQ™ was attempting to achieve, but could not scale to when used as-is.

We have (at least) two possibilities:

1. Restructure our code, modifying the parts used in CSSCDA+JQ™ in a more scaleable way — essentially, we are creating a framework of our own.
2. Migrate towards an external, hopefully better-maintained framework.

Essentially, we have a buy-off-the-shelf or build-your-own problem with their own tradeoffs — such is the nature of software.

Where I refuse to concede is that CSSCDA+JQ™ is okay, in the context of the scale of said project’s codebase. It is not.

Unfortunately, people develop Stockholm syndrome with software, too.

That, however, is the topic of another rant.

I read Getting Ahead By Being Inefficient today. It articulates what I have long practiced, and what to many, is counterintuitive.

It often means spending more time understanding the problem thoroughly, instead of deciding on a cookie-cutter solution in the name of getting things done fast. It means going against the tide of ‘obvious answers’ at times, even if you may reach the same conclusion.

I have often found that what is apparently obvious is actually not. Dig a little deeper and the proponents of said obvious solutions falter, if they are being honest with themselves.

In the article, I particularly liked the following (well-)highlighted point:

Total efficiency constrains us. We become super invested in maintaining the status quo because that is where we excel. Innovation is a threat. Change is terrifying. Being perfect at something is dangerous if it’s the only thing you can do.

Funnily enough, most people maintaining the status quo actually do a very bad job of it — mainly because the status quo may no longer be relevant. It is — dare I say it — an inefficient use of energy.

Most of my insights on topics have come from following rabbit-holes and digressing to seemingly divergent paths. At times, even analysis-paralysis has its place, because it shows that you have a plethora of choices that you can explore further. Go for a walk, a shower, shout in frustration to release the blob of paralysing energy!

In general, being inefficient in a good way, means having a holistic view of things. Generalists, by design, are never going to beat the short-term efficiency of specialists. Over a longer timeframe, however, the creative generalist has an edge.

This is not to say specialising is a bad thing, indeed, more and more I realise it’s best to be master of some, jack of many. To thrive, we must always be able to switch hats as per the demands of the situation and time.

But, mastery is much enhanced by the inefficiencies it discovers by being inefficient occasionally. I had to read the last sentence a couple of times myself, but I stick by it.

It is unfortunate, that many knowledge workers, in companies who really should know better, are treated like factory workers, when the type of work is completely different. Heck, I’d argue that factory workers that were slightly inefficient have contributed plenty to the overall production line processes!

Book Plug: If you like reading about meta-ideas and reflections, you might like a book I wrote some time ago.

Problem

I’ve just started exploring Angular 2+ and the Angular CLI. I have custom libraries in my application, and wish to avoid using stupid relative paths like ../../../common/mylib.ts in imports. This post documents what is needed to get both WebStorm (or IDEA) and Angular CLI resolving these paths cleanly.

The project structure is as follows:

// Various files not shown for brevity

 ui (-> ng new ui)
 ├── node_modules/
 ├── package.json
 ├── src
 │   ├── app/*
 │   ├── index.html
 │   ├── main.ts
 │   ├── tsconfig.app.json
 │   └── typings.d.ts
 ├── tsconfig.json
 ├── tslint.json
 └── yarn.lock

Example modules/files of interest:

///
/// ui/src/app/common/illegalargumenterror.model.ts
///
export class IllegalArgumentError extends Error {
  constructor(msg: string) {
    super(msg);
  }
}


///
/// ui/src/app/risk-reward-calculator/risk-reward.model.ts
///
import { IllegalArgumentError } from "@app/common/illegalargumenterror.model";
// Much nicer than
// import { IllegalArgumentError } from "../common/illegalargumenterror.model";
// especially for deeper hierarchies where this noise builds up.


export class RiskRewardItem {
...
  constructor(rewardFactor: Big, entryPrice: Big, stopLossPrice: Big) {
    if (entryPrice.lte(0)) {
      throw new IllegalArgumentError(`Non-positive entry...`);
    }
  }
}

Solution

We are primarily interested in the app module right now, handled by the following files:

• ui/src/tsconfig.app.json
• And the common configuration, ui/tsconfig.json; note that the app file extends this common file.

To get custom module resolution working on Angular CLI (including ng serve), we change ui/src/tsconfig.app.json, adding compilerOptions.paths (which relies on compilerOptions.baseUrl — should already be there):

/// ui/src/tsconfig.app.json

{
  "extends": "../tsconfig.json",
  "compilerOptions": {
    "outDir": "../out-tsc/app",
    "baseUrl": "./",
    "module": "es2015",
    "types": [],
    // BEGIN
    // Note that the paths are relative to 'baseUrl'
    "paths": {
      "@app/*": ["app/*"]
    }
    // END
  },
  "exclude": [
    "test.ts",
    "**/*.spec.ts"
  ]
}

WebStorm (as at Release 2017.2), however is only aware of ui/tsconfig.json (see this), and just modifying the above leaves the IDE TypeScript Language Service complaining with an error in the editor, so we also update ui/tsconfig.json.

/// ui/tsconfig.json - note compilerOptions.baseUrl and compilerOptions.paths

{
  "compileOnSave": false,
  "compilerOptions": {
    "outDir": "./dist/out-tsc",
    "sourceMap": true,
    "declaration": false,
    "moduleResolution": "node",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "target": "es5",
    "typeRoots": [
      "node_modules/@types"
    ],
    "lib": [
      "es2017",
      "dom"
    ],
    // BEGIN
    // Note that the paths are relative to 'baseUrl'
    "baseUrl": "./",
    "paths": {
      "@app/*": ["src/app/*"]
    }
    // END
  }
}

You’ll need additional entries for other modules/tests if you require, but the idea is the same.

Limitations

Initially, I found that I could not navigate to the definition of modules imported in this custom manner. Doing a WebStorm Invalidate Caches and Restart seems to have sorted this out.

Unfortunately, refactors (e.g. renames), don’t seem to be propagating cleanly. I’ve tried using both just the TypeScript Language Service, and using IDEA’s compiler.

I’ll update the post if I figure out how to get WebStorm’s key features working with non-relative/custom-resolution imports.

Alternatively, contact me if you find out!

An excerpt from the excellent (as expected, really) write-up, Toward Go 2:

“We did what we always do when there’s a problem without a clear solution: we waited. Waiting gives us more time to add experience and understanding of the problem and also more time to find a good solution”.

The whole thing is a worth a read for its clarity of writing, but I wish to focus on the importance of waiting.

A lot of technical problems are made big deals of without clear evidence of harm. If one can live with something for a while, clarity comes with time. Better solutions present themselves, and often, we get lucky: the problem can actually go away. Well, they did have to fix it in the case of Go above, but my point stands!

I recall an incident early in my career where a problem had been in production for at least a year: the company ‘policy’ was to be more secure by avoiding auto-completion (preventing auto-saved values) on form fields in the browser. Long story short, this is achieved by the autocomplete attribute in HTML forms.

A prior incarnation of the site had this feature enabled, a subsequent upgrade missed it. A tester notices it one afternoon, and suddenly management deliberates till around two in the morning, cozily from their own homes, while us developers were on standby in the office. We fixed it that night, but at an expensive human cost for no evidence of real benefit. It could have waited till the morning, at least. And arguably, saving form fields should be a choice left to the user.

Of course, this applies to life in general, too. A deliberate, active decision to wait is not the same as being complacent. It is being fully aware of the urgency of an issue, and its current lack of clarity. Not knowing, and having the luxury to wait and to defer, is a blessing. When available, gee, take it, don’t fight it!

I have used the word ‘waiting’ because of the excerpt, but taking it further, the value of deliberately taking a time to be in quietude as ways of achieving clarity should now be more… wait for it… clear.

Book plug: over the past couple of years, I have delved a bit more deeply into reflections such as these, and collected them in a book of a spiritual, non-dogmatic nature, Touching Nature. If you like applying life lessons inspired by experiences, you might like the book. I also promise that the book is more soberly edited than my sometimes rambling posts. :-)

I came across this insightful article today, and have further thoughts on it.

Diversity comes from people with differing interests, not only those who love to hack late into the night. While there are such ‘flow’ days, 4–5 hours of deep, focussed work generally is more productive and leaves one happier. Being present during a core set of hours is better than half-working through many more hours.

Really liked the example in the article where dress code does not equate to diversity or lack thereof. Though, I personally would have trouble with terribly stringent dress codes! To me it is just another form of assumption: that professionalism is not possible without dressing stringently. Certainly, dressing like a slob is another extreme.

If people are having to work longer because there’s too much work, or the work is not spread out well, that’s a deeper problem, and working longer hours is, at best, a band-aid solution.

We might lose a certain depth by diversifying, but the breadth gained in an overall more inclusive team, internally, and a product reflecting this externally often makes up plenty, and in time, a different sort of depth is gained.

Digressing a bit on working longer hours: besides looming or unrealistic deadlines, the one constant reason, in my experience has been the existence of too much non-core work (e.g. meetings!) taking up most of the day, forcing one to do actual work later in the day.

People in managerial roles who spend most of their day facilitating work through mechanisms like meetings tend to forget that developers still have to actually do the work that came out of meetings. Their day is not yet over.

I said I was digressing, but the lack of understanding by managerial staff of this phenomenon, in a sense, is also a form of accidental non-inclusiveness. It is an assumption that everyone works the same way. The coin has two sides, as usual. :-)