Here is how we compile data, analyze subjective reports, and arrive at actionable results.
Where possible, we test the printers and the filaments that we review ourselves. However, we don’t have access to every printer and filament… yet. Over time, we will update our reviews with more data about these printers and filaments as we acquire them, but for now, we have to rely on other methods for collecting and evaluating printers.
There are two main sources of data that we use to come up with our rankings. With all of this data, plus our personal experience with the world of 3D printing in general, we present you with the high quality rankings that you can use to make purchasing decisions.
Combined Community Score
The first and most important source of information about 3D printers and materials comes from the rest of the community. Even if we personally tested each material, part and printer, our experience alone would not be a sufficient indicator of the quality and consistency of any given printer. So we defer to the rest of the community for to validate our findings.
After all, for every terrible Prusia i3 clone out there, there will be a few people who have had no problems with it, or are at least satisfied, and these voices can sometimes be louder than the majority of users who have had negative experiences. In statistics, these are called outliers, and we need to control for them.
That is why we gather data from the community at large and leaders in the community specifically in order to build a more reliable, trustworthy evaluation of each 3D printer and filament. The results of this analysis are always more robust than any single experience could ever be.
The communities we tap into most often are the following:
- Youtube
- Social media influencers
- Facebook Groups
- Google+
- Forums and brand-specific message boards
After reviewing data from across these communities about a particular printer, we evaluate the community sentiment regarding a particular printer and give an overall score of one of the following designations:
- Positive (+1): community shows general enthusiasm for a 3D printer overall
- Undecided (n/a): community shows neither enthusiasm nor dissatisfaction for a 3D printer overall
- Negative (-1): community shows general disatisfaction for a 3D printer overall
But there’s a problem. What is a community is particularly large and active, with many posts by many different customers of a particular printer? What if a community has a much richer conversation than other communities? Should this community’s score be given the same weight as a less active community with only a few posts by a few customers? Of course not.
How do we deal with this?
First, we translate the sentiment evaluation for each community into an integer value as shown above. Positive sentiment = +1, etc. Then we double the score of any community that is particularly large and active in comparison to the others to represent the value of it’s sentiment. Finally, we remove the undecided scores and take the average across communities. This average represents the overall community score.
For example, if a particularly large Reddit community gave an overall positive review of a printer, Facebook groups gave an overall positive review, Google groups were neutral, and Youtube gave an overall negative review, we would run the calculation as follows:
Reddit = positive = (+1) * 2 = +2
Facebook = positive = +1
Google = undecided = n/a
Youtube = positive = -1
Combined Community Score = (1 + 1 – 1)/3 = .66
Combined Customer Review Score
After we review what the community has to say about a filament or 3D printer, we look at customer review data both on marketplaces like Amazon and also on data aggregation platforms like 3DHubs. The data from these sources is often only somewhat enlightening in comparison to the more contextually rich experiences of community members, but they often serve as a good counter point to any groupthink or “fan boy-ing” that online communities sometimes succumb too when they find a product they like.
Our top sources of customer experience review data are the following:
- Amazon
- Ebay
- Other Marketplaces
- 3DHubs
- Pinshape
Data is not always available from these sources for every printer, and each source has different metrics for aggregating customer review data. That means that we can’t simply combine the total scores from each source as if each source were equally representative overall customer sentiment. Instead, just like with the community score, we double the score of any source that has a particularly large amount of ratings in comparison to other sources
For instance, if a printer on Pinshape gets a score of 35/45, and the Amazon total customer review is 3/4 stars, we would run the calculation as follows:
35/45 = .78
3/4 = .75
Combined Costumer Review Score = (.78+.75)/2 = .765
Combined Total Pass / Fail Rating
The final problem facing us is this: how do we combine the customer review score with the community score to produce a meaningful rating that you can use to decide wether or not to buy a specific 3D printer?
Here’s what we do to give you an actionable result from all of this data: we create a pass / fail rating.
Pass = Buy it
Fail = Don’t buy it
How do we do it? Here’s how:
First, we round both averages to the nearest whole number, which is either 1 or 0. Then we take the average of these two scores. From the examples above, this is what that would look like:
Combined Community Score = [.66] = 1
Combined Customer Review Score = [.765] = 1
Printer Materials Score = (1 + 1) / 2 = 1 = PASS
But what do we recommend in cases where the community score is 1 and the customer review score is 0 so that the Printer Materials is .5?
In the section above where we gave advice about how to choose a 3D printer, the bonus tip was to make sure that any 3D printer you consider buying has a similarly positive reviews across many different sites. The reason is that it is not worth the risk of buying a 3D printer that isn’t clearly a good choice according to everyone. If there is controversy about the 3D printer or if there are an equal number of good and bad reviews across the internet, then it’s best to save your money and buy a printer that you know will work well.
As a result, in situations where the Printer Materials score .5 or undecided, we recommend avoiding that printer. So we would rate that printer as a Fail. Granted, a total score of Pass doesn’t mean that there isn’t some controversy about whether the printer is good or not. It just means that overall, the consensus sentiment across most sites is positive.
So that’s it! That’s how we calculated our ratings. It’s not perfect but works for the purposes of giving you a meaningful metric at-a glance.