[Freebase-discuss] 99% accuracy?

[Fred M. Katz]

Thank you, Tom, for a very clear answer.
It's good to know that the 95/99% number has some concrete meaning, even if
it only
addresses a portion of the Freebase activity.

I would also love to hear more from a staff member about the quality goals,
as well as any
planned approaches.  Your response suggests quite a few obvious steps that
could be taken (not easy,
just obvious.)



On Fri, Mar 30, 2012 at 11:29 AM, Tom Morris <tfmorris at gmail.com> wrote:

> On Fri, Mar 30, 2012 at 1:49 AM, Fred Katz <fmkatz at gmail.com> wrote:
> > This target is mentioned on the list from time to time.
> > Is there anything written about:
> > - just what it means: 99% true, or consistent, or complete? Or, all of
> > the above?
> > - how accuracy is measured.  Are Freebase assertions compared with
> > some other source?  Are Integrity constraints used? Do you test
> > samples?
> > - where someone could see current measures of data quality for
> > different domains.
> >
> > If you're targeting 99%, you must be keeping track of how close you are.
> > Could that information be shared?
>
> The number gets bandied around a lot and confuses information
> retrieval pros (precision? recall?) as well mere mortals like me.
> Sometimes you'll here it repeated as "99% accurate, 95% of the time"
> or some derivative, which makes even less sense.
>
> I'd love to have a staff member explain exactly what quality goals
> they have for which parts of Freebase and how they're measure, but
> here's my understanding of how things work currently (and how I
> explain it when asked):
>
> The measure only applies to new bulk data loads.  The 95% relates
> purely to sample size.  The sample size is selected to give a 95%
> confidence interval for the results.  Each instance from the sample is
> reviewed by multiple (N=3?) human reviewers who vote on whether the
> proposed change is valid.  The reviewers (by majority vote?) must
> agree that 99% of the samples are valid or the entire bulk data load
> is sent back for rework.
>
> OK, so what doesn't this cover?  It doesn't cover historical bulk
> loads.  Things loaded in the pre-history of Freebase include
> everything from crappy source data (Chef Moz) to data that was loaded
> without any attempt at reconciliation (ie everything is a new topic)
> like SF MOMA to a version of the current strategy using sample based
> quality evaluation, but with less formal quality checks (often just
> asking people to eyeball things in the sandbox) and quality thresholds
> set more by "feel."
>
> It doesn't cover recall (ie coverage).  There's no statement of "99%
> of films ever made" or "99% of films in IMDB" or even "99% of films in
> Wikipedia."
>
> It doesn't cover continuous processes such as the Wikipedia importer.
> Anything which is newly added to Wikipedia that already exists in
> Freebase will end up as a duplicate until someone discovers it and
> merges it (e.g. all the asteroids Chris has recently merged, all the
> National Register of Historic Places locations as Wikipedia catches up
> with what we have, etc).
>
> It doesn't cover data entered through the web client.  It doesn't
> cover data provided by user scripts through the MQL write API.
>
> It doesn't address systematic bias in the evaluation system (e.g.
> people confused by the UI). It doesn't address factual accuracy.  It's
> more about matches vs non-matches (e.g. this is the same John Smith as
> in Wikipedia, as opposed to we have John Smith's birth date correct).
>
> Both quality and coverage vary widely by domain, but I've never seen
> any public numbers broken down by domain.
>
> Tom
> _______________________________________________
> You are receiving this message because you are subscribed to the
> Freebase-discuss mailing list.
> To post a message to the list: Freebase-discuss at freebase.com
> To unsubscribe, view archives, etc:
> http://lists.freebase.com/mailman/listinfo/freebase-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freebase.com/pipermail/freebase-discuss/attachments/20120330/0ca5d127/attachment.htm>