[Developers] /wikipedia/en_id not giving results for redirected ids

[Shug Boabby]

Thanks Kurt!

May I suggest that for future dumps, the article.name column either
document the differences exactly, or that an additional table is
included which shows the mapping from article.name to the actual name
on Wikipedia. This all feels like black magic... I'll go and try to
reimplement in Java, but really, this is a maintenance nightmare!

I might actually just use your Python code to generate the table and
reduce the possibility of additional bugs... is the latest version
always available?


2008/7/24 Kurt Bollacker <kurt at metaweb.com>:
>
> On Thu, Jul 24, 2008 at 10:35:10PM +0100, Shug Boabby wrote:
>> and I would still like somebody to clarify for
>> me how to get the actual "wikipedia/en" name, given the article.name
>> from the WEX dumps (spaces to underscores, but what else?).
>
> Here are some python functions we often use to convert WP names to
> Freebase keys.  mql_escape() may be all you need, but cleanwikiword()
> is helpful for some messy WP names.  You may also use utf8unescape()
> when you need to handle a WP name you got from a HTTP GET.
>
> Keep in mind that not all WP names will resolve to Freebase IDs, for
> reasons such as they are not topics (e.g. disambiguation articles) or
> are brand new names that Freebase hasn't synced to yet.
>
> Let me know if you have questions.                              Kurt :-)
>
>
>
> ######################################################################
> import codecs,struct,re
>
> # Let's deal with URL escaping of UTF
> utf8decode=codecs.getdecoder('utf-8')
> isodecode=codecs.getdecoder('iso8859_1')
>
> def sub3(mo):
>    return(utf8decode(struct.pack("BBB",int(mo.group(1),16),int(mo.group(2),16),int(mo.group(3),16)))[0])
>
> def sub2(mo):
>    return(utf8decode(struct.pack("BB",int(mo.group(1),16),int(mo.group(2),16)))[0])
>
> def sub1(mo):
>    try:
>        return(utf8decode(struct.pack("B",int(mo.group(1),16)))[0])
>    except UnicodeDecodeError:
>        return(isodecode(struct.pack("B",int(mo.group(1),16)))[0])
>
> def utf8unescape(s):
>    ''' Converts UTF-8 strings that have been URL escaped
>        back into UTF-8. '''
>    # Get 3-byte UTF-8 sequences 1110xxxx 10yyyyyy 10zzzzzz
>    s=re.sub('%(e[0-9a-f])%([89ab][0-9a-f])%([89ab][0-9a-f])(?im)',sub3,s)
>    # Get 2-byte UTF-8 sequences 110xxxxx 10yyyyyy
>    s=re.sub('%([cd][0-9a-f])%([89ab][0-9a-f])(?im)',sub2,s)
>    # Get 1-byte UTF-8 sequences 0xxxxxxx
>    s=re.sub('%([0-7][0-9a-f])(?im)',sub1,s)
>    # Nuke any illegal characters.
>    s=re.sub(u'[\ud800-\udfff\ufdd0-\ufdef\ufffe\uffff]','',s)
>    s=re.sub('[\x00-\x08\x0b\x0c\x0e-\x1f]','',s)
>    return(s)
>
> # Clean up whitespace
> def cleanwikiword(s):
>    ''' Clean up the spacing chars of a Wikipedia name '''
>    s=utf8unescape(s)
>    s=re.sub('^[_ \t\r\n]+','',s)
>    s=re.sub('[_ \t\r\n]+$','',s)
>    s=re.sub('[_ \t\r\n]+','_',s)
>    s=s[0].upper()+s[1:]
>    return(s)
>
> # Do the MW hex encoding
> def dollarhex(mo):
>    return(("$%04x" % ord(mo.group(1))).upper())
>
> def mql_escape(s):
>    ''' Convert a string into a valid Freebase key value. '''
>    s=re.sub('([^-A-Za-z0-9_])',dollarhex,s)
>    s=re.sub('(^[-_])',dollarhex,s)
>    s=re.sub('([-_])$',dollarhex,s)
>    return(s)
>
> # This function shows how the above can be used together.
> def wikiurltomwkey(s):
>    return(mql_escape(cleanwikiword(utf8unescape(s))))
>
> # Do a test
> if __name__=='__main__':
>    s0='----%e9%80%81 %d4%90 %45 %1f  -_-_   blah____'
>    s1=utf8unescape(s0)
>    s2=cleanwikiword(s1)
>    s3=mql_escape(s2)
>    print ':'+repr(s0)+':\n:'+repr(s1)+':\n:'+repr(s2)+':\n:'+repr(s3)+':\n'
>
> _______________________________________________
> Developers mailing list
> Developers at freebase.com
> http://lists.freebase.com/mailman/listinfo/developers
>