[Developers] Work queues, leader boards, etc

[Tom Morris]

On Tue, Feb 17, 2009 at 7:00 PM, Jason Douglas <jason at metaweb.com> wrote:
> On Feb 17, 2009, at 3:26 PM, Shawn Simister wrote:
>>> On a related note, has there been any discussion of generalizing and
>>> opening up the end user work queues (á là the voting queue or the
>>> Typewriter queues) to allow tasks to be contributed or is the thought
>>> that each domain/application will have to replicate this?
>>>
>> I know that a couple of the Metaweb folks have mentioned this idea
>> before. I haven't seen a formal feature request in JIRA but I know
>> that the developers who built Typewriter have created another
>> project called  the Task Queue Library which seems to be what you're
>> describing. There are also two derivative projects called Genderizer
>> and Geographer which seem to be built on the same framework as
>> Typewriter.
>
> This is all still pretty experimental.  Typewriter's not exactly rock-
> solid right now if you haven't noticed :-).
>
> I think it would be cool to expose it to different domains and apps,
> but I don't think we understand at what level that makes the most
> sense yet (i.e., give me typewriter for the types in my base... or
> send this saved views to a queue to fill in a missing property... or
> here's a whole queue framework for doing completely custom work
> queues).  Definitely interested in feedback on how you'd want to use it.

I think "Freebase as game/contest" is one key method of succeeding at
the whole Web 2.0 "make money off other people's work/data" ethos, so
I'm a little surprised that these facilities aren't more well
developed.

Leaderboards are key to getting people to compete/work, so they should
be shared across applications.  I don't think you care whether people
use Typewriter or Genderizer as long as they're doing work for you, so
points should get aggregated across all apps (perhaps weighted by
difficulty of task).

Although end user points should get aggregated across apps, the
applications should have their own leaderboard so that a) users can
see which apps will be most helpful in them scoring points and b)
developers try to improve their apps.

Quality should be rewarded, so a feedback mechanism is needed whereby
both end users and applications get penalized for work that is later
deemed to be incorrect.

One general form of work is reconciliation (ie match this list of
names with Freebase topics).  For the National Register of Historic
Places data loading experiment that I'm doing, it would be great if
instead of having to choose between only allowing exact matches or
opting to create potentially duplicate topics, I could create a list
of empty slots (properties) to be filled along with the a set of known
information (name, type, etc) for each one.  Doing this in spreadsheet
form isn't the same, because then I basically have to commit to doing
all the reconciliation work myself, which isn't going to happen.

Loading simple tables is all well and good, but the real value is in
linking stuff together, which means that more powerful mechanisms are
needed to help encourage users to do this work.

Tom