[Freebase-discuss] Querying all population values

[Philip Kendall]

On Mon, Sep 06, 2010 at 06:14:12PM +0200, Benjamin Großmann wrote:
> Hello all,
> 
> We want to query all population values available in Freebase and use  
> following query:
> http://tinyurl.com/3a6hyds
> 
> We ask it the reverse way by looping all dated integers and getting the  
> according location by inverting the property  
> "/location/statistical_region/population".
> The reason is that this way we can map the results better to our data  
> schema.
> 
> The query seems to work fine but after collecting all results by using  
> cursors (with limit:100) we realized that data is missing. There are just  
> about 150 population data with values > 1 mio. and for example population  
> for city Berlin (capital of Germany) was missing at all.

Are you sure you're getting all the results? On a simple test I ran, I
got about 85k results and then a repeatable timeout when using the
cursor

eNqNj0FPwzAMhX_MLq1QNSdO7MRCE_8jyqFLN6mia6sWduDAbyeDKWxCCN7Nfk_vs9Prsk6L-O6wlzCJsx4ghlHQEzkg5tiOnWyguSzQO7SiJczrm6hLcDoLxKpfW7lNLI9NNfTj8_oyySbn0BlGLAGS4cFr0K1LYACAjDo6uMoe0ae9b3bVDZi9MVTAP7Ff_pCx534WBCpbluV31qdS2-yUobq-7yNT-hQDK7Tflv2zFFIuJVZ1HeZlyifX4dTlm-8oBh3qQiED-QlFxcJ_UTSwvVIgbjUie5cV5lnKIO9NWPMfBIp1jNsng51Wuj3sExuwnTbKs_oArEGCgQ==

I can't see any easy way round this if you _really_ need one record for
each dated integer (you're already not getting one result for each dated
integer-location), but I don't see what's stopping you running the query
in the obvious fashion and inverting in memory as this isn't really a
big data set. (You may need to be careful if you're going to worry about
a location with more than 100 population values).

As an entirely different approach to the problem, have you considered
downloading the data dump?

Cheers,

Phil

-- 
  Philip Kendall <philip at shadowmagic.org.uk>
  http://www.shadowmagic.org.uk/