[Developers] Paging through result sets.

[Nick Thompson]

there's some code in mjt that implements a seekable result list
on top of the cursor interface.  if you ask for something far down
the list it will fetch all the intermediate results from
the database until it gets to the window you want.  it caches
the earlier results so you can seek backwards without hitting
the net again.  this code was working well at one point but
hasn't been used in a while.  it makes heavy use of the
mjt.Task asynchronous framework so you do kind of need to
buy into mjt to use it.

for the less adventurous but more patient, we're experimenting
with a stored script framework that you could use to skip
forward through a list using a cursor.  you'd still be hitting
the database for all the earlier results, but you wouldn't have
to send all of the intermediate results over the wire, and you
could thin your mql responses in application-specific ways at
the same time.

     nick

Alec Flett wrote:
> 
> On Jun 7, 2008, at 4:58 PM, Arthur van Hoff wrote:
> 
>> Hi Kendra,
>>  
>> Currently you can only move forward, but you can keep all the old 
>> cursors and reuse them if you want to move back.
>>  
>> I’ve asked if the cursor could include the current offset and an 
>> approximate result set size. It would be great if you could specify 
>> the cursor offset so that you can seek through the result set at 
>> random.  This would allow you to move backwards more easily. Not sure 
>> if this will get implemented.
> 
> 
> I think this is a general problem with cursors - really they are ideal 
> for streaming data - you need to iterate all your data once, in chunks. 
> You also need some kind of guarantee that the pages don't overlap.
> 
> But most web applications need pseudo-random access - i.e. I'm on page 
> 5, jump me to page 6... or 4, or 10. Plus, web apps are probably 
> slightly more flexible to page overlaps if the data changes (i.e. if I'm 
> on page 6 of actors, and someone else adds an actor, then I go to page 
> 7, it's probably ok if I see an actor that I saw on page 6, on the 7th page)
> 
> It seems like what MQL really needs is an "offset" (similar to "limit") 
> into the set of results.
> 
> We could prevent the overlap if we also included some kind of 
> cursor-like handle (perhaps just a 'as-of-time' or what we sometimes 
> call a 'dateline' internally) to allow the results to be fixed in time 
> to when the user did the first query.
> 
> Alec
> 
>>  
>> *From:* developers-bounces at freebase.com 
>> <mailto:developers-bounces at freebase.com> [mailto:developers-bounces at freebase.com] *On 
>> Behalf Of *Kendra Kuhl
>> *Sent:* Saturday, June 07, 2008 2:40 PM
>> *To:* 'For discussions about MQL,Freebase API and apps built on Freebase'
>> *Subject:* [Developers] Paging through result sets.
>>  
>> I’m attempting to create some system of paging through result sets 
>> retrieved from Freebase. I am using a cursor to move forward through 
>> the result set, but can’t figure out how to move backwards through the 
>> result set (without some difficulty). I’ve looked through the 
>> documentation and the mailing list archive and didn’t find a solution. 
>> Does mql offer some kind of page, offset, skip parameter? Or a back / 
>> forward for the cursor? I saw the offset reserved keyword, but can’t 
>> find any documentation on how to use it or figure out if it even does 
>> what I think it does.
>>  
>> Thanks in advance,
>>  
>> Kendra
>> _______________________________________________
>> Developers mailing list
>> Developers at freebase.com <mailto:Developers at freebase.com>
>> http://lists.freebase.com/mailman/listinfo/developers
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Developers mailing list
> Developers at freebase.com
> http://lists.freebase.com/mailman/listinfo/developers