I got a devastating critique from one of my beta readers for my book draft “Interpreting Machine Learning Models With SHAP”.
The comment was along the lines of “doesn’t read like a book yet”. More nuanced and friendlier than that, but that was the gist of it.
The feedback catapulted me right into the 5 stages of grief (exaggerating a bit here)
The draft can’t be that bad, because other readers liked it (Denial)
The feedback was too harsh (Anger)
Maybe I can make some light adjustments to the chapters and it’ll be fine (Bargaining)
I was ready to publish this book … does that mean I’m a terrible writer? Maybe I should give up on writing altogether (Depression)
The reader was right, the flow of the book needs to be improved! (Acceptance)
The feedback put me in the position where I had to decide: just ship it? Light edits to improve the flow? Or dissect the chapters and rebuild the narrative structure, which would mean a delay in the publication of the book?
Accepting feedback
In light of the feedback, I reassessed what I had written and also read the other reviews again. I came to the conclusion that the flow of the book wasn’t great. This meant for me: a lot of rewriting, adding and dissecting sections, throwing some text away, and changing the order of content.
It also means that the publication will be delayed. I expect to publish the book now at the beginning of August.
To give you an idea of some of the problems the book had and how I tackled them:
If you had no idea what SHAP was, the book left you hanging for quite a while. Now the introduction features a short example.
Early in the book was a “Getting Started with SHAP” chapter with code and all, but before any of the theory was explained. Then came theory, then again application, which made the flow of the book weird. I got rid of the Getting Started chapter and separated theory and application.
Multiple reviewers suggested switching the order of chapters. Now the order is, in my opinion, much more logical.
The theory chapters were missing parts of the theory and in hindsight, I didn’t like how the theory chapter was structured. Now I separated theory into 3 parts: 1) game theory for original Shapley values, 2) how these concepts are transferred to explain machine learning predictions (SHAP), and 3) why and how SHAP is estimated.
I’m super grateful for the feedback of all the readers and I’m much more happy with the flow of the book now.
How the flow of the book got stuck
At some point, I had decided on the initial flow of the chapter, which, from today’s perspective, I’m no longer happy with. I did some digging and it became clear to me what the reason was:
Laziness.
One of the reasons why I decided to write the book on SHAP was that I already had a ton of material: Two chapters in my Interpretable Machine Learning book and multiple blog posts — big blocks of text that are more or less self-contained. I broke them apart a little but mostly reused them. I rewrote the blocks somewhat so that the parts fit together.
But these large blocks clocked the flow of the book because I didn’t logically build-up, for example, the theory of Shapley values. Instead, I focused more on placing the big blocks that I already had and then tried to connect them and build around them.
Writing around big chunks of text is like planning your living room around some pieces of furniture that are misplaced in the room. You struggle to come up with a great interior design because you falsely assume the bulky pieces of furniture are given/unmovable.
Lessons for technical writing that flows
I’m taking away some lessons and improvements. Maybe they are useful for you as well:
Collaborate with beta readers and truly heed their feedback. Beta readers are the closest to getting an impression of what the experience will be like for your future readers. Invaluable feedback that you otherwise wouldn’t get.
I decided to do another round of beta reading. If you are interested to get early access to the book and provide feedback, send a mail to readers@christophmolnar.com
Work with an editor. Something I want to explore for future projects. An editor can help spot these types of problems as well.
Separate theory, implementation, and application in your writing. Not necessarily a separation by chapter, but it should be as clear as possible when you refer to which of the 3 things. For example, “sampling 5 times” could be based on a paper (theory), a default choice of the library (implementation), or an example that you chose (application).
Work with atomic notes. The reason I ran into the problem with the chapter flow was that I reused too big chunks from what I already had and that clocked the flow of the book. Don’t get me wrong: Always build on what you already have. But the smaller the building blocks, the better you can mold them into a good narrative.
Add learning goals to each unit (if that makes sense for your content). This forces you to be intentional about the content, giving your readers clarity.
Also, keep in mind: Done is better than perfect, so it sometimes can also be okay to just ship it. Depends on your priorities and good judgment. In my case, the decision was to reshape the book and make it better.