Analysis of SEOmoz’s Published Data on Ranking Correlations, Latent Dirichlet Allocation (LDA), and Spearman’s Correlation Coefficient. Google Engineers are Sleeping Well at Night.
Abstract In this article we discuss confidence intervals from sampled data sets and analyze the published statistics from a recent SEOmoz article. We conclude that from a “particular data set”, in this published experiment, we can confidently conclude nothing about the relationship between calculated Latent Dirichlet Allocation (LDA) values and the ranking of a web document’s position in the commercial search engine’s result. The published data forces us to conclude that for any particular search query the SEO practitioner is not sure if a web document should have a low or high LDA number. Using Matlab simulations and the published data we show that the standard deviation reported for the “mean Spearman’s coefficient” is nearly equal to the average coefficient derived from the data sets. To interpret SEOmoz’s data correctly a scientific presentation of the results would include, at a minimum, an assumed confidence level and the calculated number of degrees of freedom.
Analysis of Published Data
After discussing published data from SEOmoz that was referenced on Dr. Garcia’s blog, we are in agreement that the statistical results published looked incorrect and many were asking for more information (i.e. published data). We do not need to see all the data to show the misapplication, but it is helpful so that we can see if the distribution is two-tailed, one-tailed, monotonic, etc. Publishing the exact method of calculation is usually needed to verify the results of a measurement. A colleague emailed this post from SEOmoz. The post is a correlation study using Spearman’s coefficient and LDA calculations. The formulas for calculating LDA are found in this paper by Blei et al. (2003) Techniques for gathering sample sets are described in the paper: “Distributed Query Sampling: A Quality Conscious Approach.” In this article we will focus on the techniques used to analyze the sampled data and not the chosen sets of data.
The statistical formula used in the SEOmoz’s article is Spearman’s rank correlation coefficient. The formula is straightforward but the application and interpretation of it can result in misleading conclusions. Spearman’s coefficient is calculated as:
Where r’ is spearman’s coefficient, N is the size of the sample set, and d is the statistical rank of corresponding variables. At issue is the significance of the published coefficients, the sample sizes, and the validity of reporting a mean from calculated correlation coefficients. It is well-known that correlation coefficients are not additive. Meaning we can’t normally take samples for DIFFERENT experiments and total the correlation values. In this case the sampling that is taking place is from the SERPs and the results are generated form a list of “random” search queries. The sampling for each set (search phrase in which we are calculating LDA), in SEOmoz’s latest experiment, appears to be based on sample sets between 6 and 10.
When examining the values we see in the the published data set we see a two-tailed curve because there are positive and negative values. We can also see some values that are less than zero (negative). A good negative correlation means that an increase in one variable causes a decrease in the other variable. In this case it means that web documents with lower LDA scores are “mostly” ranking above documents in SERPs with higher LDA scores. Following are the possible outcomes from the numbers that result from Spearman’s coefficients:
1. No correlation between the variables
2. Positive correlation between the variables (indicated by positive values).
3. Negative correlation between the variables (indicated by negative values).
When we perform a sample of a data set we must deal with that data set. We cannot create new data sets, showing new correlations and add them to previous data sets. Typically, the minimum number of samples needed to create a Spearman’s coefficient is 4. However, if we create a new search query we have created a new data set and we must analyze the results of that query and determine the correlation on that data set. If this is not the case we would need mathematical proof of this using the known properties of Spearman’s coefficients.
Once we take a sample and calculate Spearman’s coefficient, we then use critical value tables to determine the significance of the sample. See Table 1.0, critical values:
In Table 1.0 (for a two-sided distribution) n is the number of samples we have taken for the CURRENT set we are looking at. So if we have 8 samples (i.e. 8 search results), we must have an r’ of .738 (the column under 5%) to be confident that 95 times out of a 100 the data occurred because a relationship exists and not because of pure chance. If we have 8 samples we can see in table 1.0 that to be confident 99 times out of a 100, the data occurred because a relationship exists and not because of pure chance. Notice that with only 4 samples we can’t achieve the 5% confidence level. Most of the samples in the data sets appear to have 8 samples. Technically, n in table 1.0 is indicating degrees of freedom.
In the study done by SEOmoz there is a low sample set for each search query. On average most appeared to have 8 or less. This means (not factoring in degrees of freedom) we need Spearman values around 0.75 to be 95 % confident that our measurements did not occur by chance. These numbers tell us only how confident we are that there is a relationship between the two variables. R values around .33 with only 6 to 8 samples are considered very weak relationships. It is up to the observer to define reasonable confidence intervals. When comparing to other papers one also has to examine the number of samples taken in those studies. In the Stanford paper referenced by Garcia (an example referenced as a case for publishing extremely low coefficients) they are stating that 0.12 is a very low correlation, not a high one. The following graph shows that our required “significance level” depends on the number of samples in the data set. As we approach 80 samples the correlation coefficient required to put us at a desired significance level drops.
Shown below is a histogram of the published data showing the mean and the standard deviation being about equal.
Using Matlab we created the graph below which shows the Average LDA Ranking versus the position in the search engine. This plot uses ~61 different data sets. It shows an obvious skew for LDA values and the number one position in the SERP. The other positions seem unaffected by the LDA values.
Looking at the data from the report we see a specific example for the search query “dining room sets” the published Spearman’s coefficient for this sample set of 8 documents is -0.619. Table 1.1, below, shows the published values from the study:
This means that for this “particular” query “dining room sets” we see that documents with lower LDA values (.43 is the LDA value for overstock.com’s site) ranking higher than other documents, and we are fairly confident (Spearman Value of -0.619) that, if we want to rank higher on this search phrase, we should have a lower LDA value than a high one for this particular search query. If we choose to discard this sample of data, along with all negative values, we will skew the mean of the coefficients to higher positive values.
After a sample set is taken Spearman’s value should be calculated on the current data set. If the value is not within the confidence interval, the hypothesis is rejected. In previous discussions, the referenced papers are averaging the final results of previously sampled data sets from acceptable data sets that met required confidence levels. Not averaging data sets that produce significantly low coefficients and negative correlations is an attempt to make a “mean coefficient” positive.
Calculation of the Spearman’s Coeffecient Using Tied Ranks
If we were to assume that we can use the published data from SEOmoz as coming from a “single system” (not exactly true but is in effect what has happened by averaging all their values) and that all search queries are treated without particular bias, we expect to be able to use so-called “tied rank” calculations to arrive at a correct Spearman’s coefficient for the entire data set. To do this we could assume about 6 samples per search query and there are 555 sample sets (not all sets had the same number of data points). Using “tied rank” calculations we have to calculate new rank positions. To do this one sums the rank positions and divides by the number of tied cases. To do this calculation one would have to determine the number of degrees for freedom. The formula for Spearman’s coefficient using tied ranks is:
If we can treat the outputs of all sampled sets as coming from a “single system” then we should be able to take all the data and arrive at a coefficient before calculating each Spearman’s coefficient independently.
Twitter
Facebook
We compute standard error of the mean. That is included as error bars in the chart used in the post, and you can recompute it from the data I provided.
Standard error of the mean is sufficient to see there is extremely high confidence in the results the post showed. You can see we are confidant that the mean spearman correlation coefficient for the LDA is higher than that for TF-IDF, and unique linking IP address. You can also, of course, see we are quite confident it is higher than zero.
Using standard error of the mean to show confidence avoids several of the drawbacks of what your post suggests we should have done. As you note, because of a shortage of degrees of freedom for any given SERP, computing confidences of the correlation coefficient for a specific SERP is not very informative. Pooling the pair data points of results across SERPs would make the sample no longer qualify as a “simple random sample.” It also intuitively fits with what we are trying to show - that conclusions from the computed sample mean generalizes to a larger population of queries. You can read a complete discussion about why I prefer the methodology we use to measure correlation and show statistical confidence here:
http://www.seomoz.org/blog/statistics-a-win-for-seo
In the post and comments by Dr. E. Garcia that you link to, he makes or relies on several positions that are contrary to establish mathematics, including:
1) The mean correlation coefficient is uncomputable. He is wrong, it is computable.
2) Spearman’s correlation measures any monotonic correlation just as Pearson’s correlation does. He is wrong, Pearson’s measures only linear correlation.
3) PCA is not a linear method. He is wrong, it is.
In my post I referenced above, I go through these and other incorrect statements in greater detail, and cite/quote them directly.
Let’s keep this as short and as simple as possible and it will be easy to see the issue. Here is how the experiment was done in layman’s terms so that others may follow:
1. You had a list of search queries
2. You took the top 6-8 results that resulted from the search queries in 1.(see data) This is what is known as a SERP (search engine results page).
3. You calculated the LDA and a correlation (how much LDA “influences” rank) for each sample set.
4. You then averaged the correlations from all of these sets together.
The first place to start is with the statement you just made:
Note: (degrees of freedom, below, is basically samples and is about 4-8)
Your comment:
” because of a shortage of degrees of freedom for any given SERP, computing confidences of the correlation coefficient for a specific SERP is not very informative.
We can follow the conclusion quickly from here. Ignoring everything else for the moment.
1. If, for any given search result (generated from a query) the data is “not very informative”, as you just pointed out, then each calculation you made using Spearman’s formula , is “not informative”. You calculated Spearman’s coeff. for each search query you performed (about 500 different search phrases) and then used the top 6-9 results. If each individual sample that you took is “not informative” then summing the correlations from each “non informative” sample and calculating a mean is even less informative.
Now, by summing them all up together you, in effect, violated your second comment:
“Pooling the pair data points of results across SERPs would make the sample no longer qualify as a “simple random sample.”
Since you averaged all the correlations for each “non informative” sample together you HAVE basically pooled all the samples you took by summing their coefficients together instead of the data. According to your comments you just violated what would qualify as a “simple random sample”.
I would say that the experiment needs to be redone correctly to get anything useful out of it.
I think your mistakes are pretty straight forward to see.
Your claim is that because we are not confident about the ability to generalize information based on the coefficient from a single SERP, we cannot have confidences that the mean of hundreds of these coefficients can generalize. This claim is incorrect.
Consider a more common example. Although the length of one man’s life may not be very informative about the average lifespan of the population, the average of one hundred men’s lifespans is.
It is quite common for a single trial to not be “very informative” but for the mean of hundreds of trials to be quite informative.
….
Regarding the second point about simple random samples.
I did not say any possible way of combining data across SERPs would produce something that wasn’t a simple random sample. I was saying specifically what you suggested wasn’t. Other options, like what I did, is rather good at qualify as a simple random sample for a meaningful population.
To see this, consider the larger population of data points we are trying to generalize to - different SERPs. If immediately pool all results from all SERPs before calculating a coefficients, consider what you have. If you have one trial (paired data point) from a given SERP, you quite likely to have 4-7 other trials from it, where as for SERPs in the larger population but not in our sample we have 0 trials from. It is pretty easy to see that the trials have not been selected independently of each other in a way sufficient to qualify as a simple random sample.
This contrasts to my sample, which only includes a single coefficient as the trial from each SERPs, and so cannot have the problem of losing the independence of trials from having a disproportionate number of trials from any one SERP.
I am surprised you would use this example. Here is why.
When we calculate average life span of a population we are taking a KNOWN quantity, when they die, and then averaging everything together. This is fine. We end up with the average life expectancy. If it turns out to be 80 it means nothing about the cause of the death. Death is influenced by many different things. If we average up when people die then what we report is correct. The average age at which a population dies. I do not know what I need to do to live longer.
If we applied this technique to ranking web documents on Google here is how you would do it. We would ask every website owner where their home page ranks on Google (known quantity). Then we would average them all together and conclude the average website owner ranks in position X on Google. Not very useful but a true number. Not sure what I can do with the information as a website owner.
What you are doing is trying to determine a cause for why a web document is ranking in a certain position. Which would be like taking your simple example and determining the cause of death. Much more complicated.
You are operating on a black box, Google, and trying to estimate if LDA and ranking position have any strong correlation. If you prove that they do you still can only say one thing. We are X percent confident that LDA influences ranking position.
I would run the following experiment and see what happens to your data. What you will find is no convergence between what you have now and the new data.
Take the first 100-200 ranked documents (instead of 6-8) and see what correlation value you get. Or use Kendall’s tau and see what happens.
I encourage you to publish this data. Possibly investigate using Kendall’s Tau as well to see if there is convergence.
You suggest that unlike a person’s lifespan, the coefficients that we are averaging over are unknown values, and this somehow invalidates our use of the mean. But that isn’t right.
The coefficient for the top results of a single SERP is a known quantity. We computed it with a lot of precision. So we are taking the mean of known values.
You can point out the coefficient we compute is an estimator for unknown values, such as what the coefficient computed on results with a larger population of SERPs, or the coefficient computed on more results from the same SERP. But this is the same for one person’s lifespan. One could view a person’s lifespan as an estimator for the unknown value of average human lifespan, or the average lifespan in his country. But in both cases, the values themselves (the measured lifespan or the computed coefficient) that we are averaging over are known quantities.
….
You bring up several tangents we could discuss. Namely, to what degree does correlation implies “influence,” or what are the merits of the alternative methodology you suggest. But let’s try to reach some conclusion about the validity of the methodology I used first. I think we have your objection in precise enough language that a definitive answer about your objections correctness is straightforward.
I can’t agree with this. You are calculating correlation not averages. You averaged the correlations. Yes, I would view a single person’s life span as a data point to help find an overall average of a populations life span.
However, average lifespan says nothing about a possible CAUSE (or correlation) on the lifespan.
If you wanted to publish how, on average, a site ranks after having a certain LDA value that would be valid number and it would take a lot of sample points. The number, however wouldn’t be that useful.
I would do the experiment I mentioned and report the correlation numbers. I think it would help you see what is going on and give you another reference point to compare with.
Great discussion.
I could clearly follow as your discussion led towards finer understanding of Ben’s experimental methods, and what they may or may not actually show. In the explanations and back-and-forth addressing of points made, I could also “see” much of the perspective the investigators carried into their efforts and data analysis.
Perspectives are often clues to bias, and access to perspective is very helpful for interpreting published works. That’s why scientists work in community (peer review, published abstracts, meetings, and often “letters” or emails these days).
It’s a shame such discussions don’t take place *before* results are published to support marketing claims. Perhaps one day society will advance to the point where we separate objective investigations from commercial marketing, and smart people can freely follow their inquisitive minds free of for-profit pressures. I know Ben was paid to do his work, and I assume Sean took a loss when he directed his Engineers to re-analyze Ben’s data for the benefit of the search community.
You are implying there is some conflict in a value being an average and also communicating meaningful information about the distribution of correlation. There is no conflict.
We published average correlation coefficients. They really are averages. They also communicate meaningful information about the amount of correlation.
Average lifespan is meaningful information about lifespans.
Average correlation is meaningful information about correlations.
@John:
I agree my back and forth with Sean has been rather productive in terms of figuring out precisely what he objects to and working out how valid this objection either is or is not. Unfortunately for me, the vast majority of folks who read his blog will not see this discussion. Most folks who will ever read it already have, and from it they will have gotten the impression our results were not mathematically sound.
As you suggest, it would be nice if people would work out the validity of claims before they publish them. I have discussed the merits of using mean spearman’s correlation with a considerable number of people, and even published on our blog a lengthy discussion of how and why mean spearman’s correlation is a great methodology for this sort of work. I published this lengthy discussion of methodology prior to publishing the LDA statistics. Because of this, I published those statistics with high confidence in the validity of our methodology. I suspect that most folks who published claims that our methodology was invalid were less careful.
I would be intrigued to know what you view as marketing claims. I released some correlation statistics, data, and a free tool so people can generate more themselves. We do not currently sell any tool that uses any of my results. We do not make money from our blog any more directly than Sean makes money from his (he sells a journal, we sell tools, and our blogs only indirectly help either). So I’m not sure how publishing our statistics and methodology on our blog is any more a marketing claim than someone who publishes claims that we did something wrong.
Although I know it sounds like I am disagreeing with you, I really am not. Our industry definitely has a problem sorting out credible claims from non-credible claims. I can only hope as more people start crunching numbers and publishing results, we can move to the more collegiate model you describe. As it is now, the care behind a claim or counter-claim is not very related to how widely it is disseminated.
In my experience when people publish average correlation coefficients it typically comes from someone that has gathered a lot of previously VALID statistical studies and then said “On average the correlation coefficients for these studies are X” This would not then adjust the correlation coefficient for any particular study. People publish things all the time that are not correct and even if it is in a journal, especially the softer sciences, there are mistakes and things that don’t really hold any value.
I am not sure if you read Dr Garcia’s tutorial on this yet:
http://www.miislita.com/information-retrieval-tutorial/a-tutorial-on-correlation-coefficients.pdf
They also give some guidance on what a strong correlation is. In that paper he went and consulted with other experts.
We can average correlations together as much as we would like but if the correlations themselves are not good I think we get even less info by doing averaging. Averaging is not really used to give us a indication of how well one variable influences another. I would move away from thinking about averages because averaging doesn’t give us strong correlation data. It gives us averages.
I would put together a mathematical proof that your technique is valid and present it so that we can look at it.
Or you could do some of the other experiments that we are suggesting.
I would like to make a note on the averaging of correlation coefficients.
“Because the value of the correlation coefficient is not a linear function of the magnitude of the relation between the variables, correlation coefficients cannot simply be averaged.” quote found at http://www.statsoft.com/textbook/basic-statistics/
The function of the magnitude of the correlation coefficients is concave up in the region that most of the data resides. We can use Jensen’s inequality to determine that the actual average will be less than or equal to the calculated average.
To avoid this issue that data should first be transformed using a Fisher transformation and then averaged.
Dr. Garcia’s tutorial is not a credible source of information, and was written after an odd evolution of positions whose only common thread was to claim that something we did was wrong. He initially claimed that to compute standard error for our mean spearman correlations we needed to do so as though it was not a mean. That was wrong. Then he claimed straight up that one could not compute the mean of a set of spearman correlations. That also also wrong. The position of the tutorial appears to be that a mean spearman correlation is computable, but that doing so without using the fisher transform to weight them is always invalid. He is, of course, wrong again.
I can point out where in that tutorial Dr. E. Garcia is again making statements contrary to accepted mathematics, although he will probably just roll on to making new claims and never retract his old ones. But what the heck, let’s do it.
Dr. E. Garcia states “since correlation coefficients are not additive, it is not correct computing mean correlation coefficients by averaging individual values.” And that isn’t a small point for him - that really is the crux of his argument in the tutorial.
If one checks page 82 of “Meta-Analysis of Correlations: correcting error and bias in research findings” it asks the question “is the weighted average always better than the simple average?” In then cites an example where the unweighted analysis does better, and notes for another problem that “the Fisher z transformation produces an estimate of the mean correlation that is upwardly biased and less accurate than an analysis using untransformed correlations.” So sources more credible than Dr. E. Garcia’s certainly disagree with his categorical statement that “it is not correct computing mean correlation coefficients by averaging individual values.”
But the larger problem with his point is that I’ve never used mean spearman correlation as an unbiased estimator for what the correlation coefficient would be if I pooled data points first. Perhaps he was, and that is why he made his initial claims we should compute the standard error of our mean as though it was not a mean, but that would make him who was wrong, not me.
Simply taking the sample mean as we report it as an estimator for the population mean of coefficients across a greater number of serps is how I recommend interpreting our statistics, and as an estimator for that it has no bias. It is meaningful, and avoids all of the theoretical and practical problems about either pooling the data points or using a weighted mean.
….
Another inaccuracy in his tutorial is with regards to the amount of bias from using the Fishher transform. Replacing instances of 1 with 0.999999 does not mean one will have a 99.999999% accuracy in terms of producing an unbiased estimator. Values near 1 have issues, not just the value 1. As values approach 1, there becomes a lot of upward bias. Given the few of degrees of freedom behind the coefficients for each query, we will get non-trivial upward bias in correlation coefficients. And unfortunately the bias won’t be consistent as it will depend on the number of extreme coefficients in each sample. It would be very hard to trust results computed as he suggests.
….
Regarding your point about the “correlation themselves are not good,” I would refer you to the argument I made above. The coefficients I am averaging over are extremely accurate coefficients for the correlation of the top results of that SERP, as we computed it exactly. It is only “not good” as an estimator for the mean coefficient for the population of the tops of a larger population of SERPs. But it is hardly a surprise that a single value can be a bad estimator for something, while the mean of many of them is a good estimator for it.
@aaron
The quote from statsoft can be a bit misleading. I wrote a blog post that discussed that specific quote, what it is meaning, and why it doesn’t contradict what we did. You can find it under part B of rebuttal 1 in this post:
http://www.seomoz.org/blog/statistics-a-win-for-seo
The confusion comes from how they are using the word “average” in that context.
@Ben said “I would be intrigued to know what you view as marketing claims”
The only reason I know of your LDA work is because of the claims that were broadcast by executives and employees of seomoz, and re-broadcast by others during and I believe after the seminars. I have no other avenue to have heard them: I don’t follow seomoz, am not a member, don’t regularly follow any seomoz employees.
I think the claims were remarkable enough to be restated widely. I recall suggestions of industry-game-changing developments and the like, tied to the seomoz brand. Since seomoz sells tools and competes for attention in the search marketing space (to draw eye balls and convert them into support for market authority, paid subscribers, etc) they are marketing claims.
There are more fastidious people than I do ask for copies of the claims… I don’t tend to pay a lot of attention to them. I do believe seomoz has positioned itself such that it needs to be very careful about suggesting “research shows”. Hopefully that is a side effect of earned respect, not history of misleading claims.
By the way it is worth noting that only paying members would know that you do not have a tool inside seomoz tools based on these revelations. Observers who are not on the inside (who are also your best prospects for conversion, e.g. the priority “targets” for your marketing) are likely to assume that your developments, discoveries, insights, research, etc. support the tools available to your paying subscribers.
@John
There is something to what you are saying.
SEOmoz spends time looking at this stuff because in the future it will help our tools, and we release our findings in the mean time in part because it helps our brand.
So there is a sense that it is fair to call it marketing, and more so for us than it is for Sean. My statement to the contrary was not fair.
I work hard to only make any statements that I supported by evidence. I (incorrectly, it sounds like) read into your remark about “marketing claims” and your point that SEOmoz pays me a salary to suggest otherwise.
Looking over what you wrote again, I believe I took offense unnecessarily.
@Sean
I’m waiting for your explanation.
Your blog post does not claim that you are unsure if our methodology is correct. It claims to be able to show it is incorrect - to show that there was a misapplication of data.
From our back and forth in the comments, we have gotten the reason for your claim narrowed down to “we can average correlations together as much as we would like but if the correlations themselves are not good I think we get even less info by doing averaging.” Unfortunately, these words do not have a precise statistical meaning.
I have offered a reasonable interpretation of your statement that is more rigorous. I believe by “not good” you are referring to the accuracy of the value as an estimator for the expected value of correlation coefficients on a larger population of SERPs.
I have also pointed out with this interpretation you are wrong.
To stand by your post, you have two reasonable options:
1) Show that your claim, as I have interpreted it, is actually true.
2) Provide an alternate rigorous interpretation for the claim that is both true and still conflicts with the claims we made on our blog.
@Ben I don’t take anything personally unless is is a falsehood aimed to discredit me or otherwise harm people. Those I take very seriously. Ask Rand.
As a concerned scientist you might take a look at Ben Cook’s post http://skitzzo.com/archives/seomoz-hype-machine.php He re-published many of the public statements from your boss and executives, which they made while you were at the podium, away from the meeting, or otherwise engaged in the real work. Good examples, I think.
Personally I care more about claims expressed in SEO article titles that live forever in Google. They do harm long after they are published. Here’s one of yours that states unequivocally that LDA tracks Google Rankings:
“Latent Dirichlet Allocation (LDA) and Google’s Rankings are Remarkably Well Correlated”
One from last year, which was completely debunked by the community, remains online and ranks for SEO searches even today (it has a disclaimer on it that it is, in fact, incorrect): “Tests Show PageRank Sculpting with Nofollow Still Works”
I’m not asking you for anything, just noting what I see that is related to your work and your company’s use of your work.
@Ben We are claiming that we can’t conclude anything from your published data. To make an interpretation you need to know what you are stating as your number of degrees of freedom.
In my abstract I am asking for a number of degrees of freedom and a confidence level. This number needs to be published with statistical data of this type.
Rather than confusing the issue with so many different questions it is easier to get one question answered at a time.
We also see a skew in the calculated LDA value for the first position. You stated in your excel data that:
“The dataset below is not the same as the one the chart was made from. It does, however, show comparable results.”
Not sure if you are referring to the calculated LDA values in the spreadsheet? If so we need to see the actual values used to calculate your published value of 0.35.
We need your stated degrees of freedom. Ignoring everything else for now, you should state your number of degrees of freedom and what your desired confidence level is.
For anyone interested, it appears the correlation is actually half what SEOmoz initially claimed - http://www.seomoz.org/blog/lda-correlation-017-not-032
Sean, you’ll be pleased to know that Ben still says you’re wrong despite his mistake :)