Problem Context and Hypothesis
- Unlike other comparable binary features (Like/Upvote), Claps’ non-binary nature (1 to 50 claps) accentuates the problem of social proof and leads to clap inflation
– Unlike most other common feedback metrics (such as Likes on Facebook or Upvotes on Quora), the Claps feature is non-binary in nature and can range from 0 to 50 (A reader can give a maximum of 50 claps to an article on Medium).
– As human beings, we are influenced by social proof; in Medium’s case, a reader’s action of clapping may be influenced by existing number of claps by other readers on the article. This by itself, is not a unique phenomenon. However, the non-binary nature of Claps can accentuate the problem of social proof and lead to clap inflation.
- Beyond a certain number of claps/article, there is an:
(a) increase in reader’s propensity to clap the article, and
(b) increase in extent of appreciation (or the number of claps) for that article
- Clapping behavior may be inconsistent among readers
– Claps helps in expressing more appreciation for the author’s work as against a plain vanilla metric.
– However, its non-binary nature also lends itself to a few problems. Let’s say (a) two different readers really like an article, and (b) they clap 15x and 25x, respectively. This may not necessarily mean that the first reader likes the article more than the second one — the first reader might be more judicious in his/her appreciation as compared to the second reader.
- For a particular article, different readers with the same intent of appreciation, may express their appreciation (via number of claps) differently.
- Biases: The reader might be biased to read the article or appreciate the author’s work upon looking at current number of claps on the article
- Quality: For a reader, it is difficult to discern the true quality of an article as it is subject to clap inflation.
- Content Discovery: Discoverability of article in search results can be heavily influenced (*subject to the importance accorded to the number of claps in the Medium’s search algorithm).
- Reader Feedback: As mentioned in Hypothesis #2, it is difficult for an author to gauge the positive reception of his/her work, as different users would tend to use Clap differently. On a separate note, if a user momentarily rests his finger on the Clap button more than what’s required, does that mean that the user likes the article more? Highly arguable as there is a lot of friction involved in undoing the action of claps.
To further investigate the hypothesis identified above, the following data points or metrics might be worth evaluating:
- Difference in clapping behavior across different platforms (Desktop vs Mobile): In order to evaluate the difference in clapping behavior, both Desktop and Mobile platforms would have to be evaluated separately.
– Mobile: The number of claps are shown right at the end of the article. Moreover, as compared to other information elements on the screen, the total number of claps have not been accorded a higher importance.
– Desktop: Unlike the mobile view, the Desktop view shows the number of number of claps upon immediately opening and scrolling the article. As against the Mobile view, the the number of claps have been prominently shown right on top of the primary action button, i.e. the Claps icon itself.
Note: All the data below would have to be dissected further by Desktop and Mobile
- Propensity to clap the article: To gauge this we would at 2 metrics, (a)average percentage of article before first clap and (b) fans/unique reader visits. Details below:
– Metric#1: ‘Number of Claps/Article’ plotted on x-axis and ‘Average percentage of article read before first clap’ by the reader plotted on y-axis.
– Metric#2: x-axis: ‘Number of Claps/Article’ plotted on x-axis and ‘Fans/Unique Reader Visits’ by the reader plotted on y-axis.
Note: For the uninformed, the fans in Medium’s context refer to people who actually clap or appreciate the article. Hence, fans form a sub-set of unique reader visits.
- Extent of appreciation (or number of claps) for each article: Number of Claps/Unique Reader Visits or Fans can serve as a good metric for this. Details below:
– Metric#1: ‘Number of Claps/Article’ plotted on the x-axis and ‘Average Number of Claps/Fans/Article’ plotted on y-axis.
– Metric#2: ‘Number of Claps/Article’ plotted on the x-axis and ‘Average Number of Claps/Unique Reader Visits/Article’ plotted on y-axis.
In this case, we would look at various frequency distributions aggregated over thousands of articles. If Hypothesis #1 (a) were true, then this should be a steep upward sloping curve beyond a certain number of claps/article.
- Distribution of number of claps: For analyzing distribution of number of claps, I would evaluate the following 2 cohorts: (a) Articles with <100 claps (b) Articles with 1,000+ claps (Note: The number mentioned in each of these cohorts can be changed based on observing actual data).
– Metric#1: Aggregate clap distribution data of ‘Average Number of Claps/Fans/Article’.
– Metric#2: Aggregate clap distribution data of ‘Average Number of Claps/Unique Reader Visits/Article’.
To analyze the 2 metrics mentioned above, there will be ten clap intervals ranging from 0–5, 5–10, 10–15…45–50, plotted on the x-axis; the percentage distribution will be plotted on the y-axis.
(Including the ones that Medium has possibly already implemented)
- Conduct follow-up AB experiments
– The above-suggested data points can carve out interesting insights about the difference in behavior of Desktop vs Mobile user. There could be AB experiments designed to figure out the (1) right placement of claps icon and (2) the prominence and placement of the existing number of claps on a particular article.
- Finding the value of a clap, unique to every user
– Since different readers clap differently, it is important to find out how valuable is a reader’s clap by analyzing the clapping behavior for the same user over a period of time. I can take claps/article read as the key metric here.
– As an example, I would assign a higher value/clap for a reader who only claps 0.5/article, as against another enthusiastic reader that provides 2 claps/article.
– It seems that this is already implemented by Medium as mentioned by some of the other available articles available on Medium on the same topic.
- Limit the non-binary nature of claps
– Instead of providing the option to clap from 0 to 50, it might be prudent to limit the upper cap of claps/article to a much lower number. This will help in limiting the non-binary nature of Claps as a feature — thereby, potentially reducing the Claps inflation problem highlighted previously.
– However, the decision around limiting the upper cap on claps should again be governed by data — in this case, distribution of number of claps (details of which have been described in the ‘Data’ section). For example, the data might suggest that 85% of the claps are below 20 claps. In this case, limiting the upper cap might not be an important problem to attack.
– The reason why this might not be implemented by Medium is that this solution goes against the very purpose ‘Claps’ as a feature might have set out to achieve — to provide a creative and intuitive way to appreciate an author’s work, mimicking how we appreciate offline.
– Additionally, current UI/UX would not be suited to such an alternative, as it is currently fairly simple to give higher claps and even accidentally clap higher than intended. The design would have to be re-thought to have a more deliberate intent by the user while assigning these claps.