From mt at alum.mit.edu Sun Apr 1 00:46:19 2007 From: mt at alum.mit.edu (Mike Travers) Date: Sat, 31 Mar 2007 17:46:19 -0700 Subject: [Developers] bug: upload fails for certain strings Message-ID: [Apologies if this isn't the right place for bug reports.] I'm bulk-uploading some data and ran into a bug. The upload service fails for certain strings. Here's what I see, using the Python upload routine included in the API documentation: >>> metaweb.upload("random string", "text/text", credentials) u'#9202a8c04000641f800000000504e188' >>> metaweb.upload("P1,", "text/text", credentials) {u'status': u'400', u'messages': [u'image/x-portable-bitmap is not an acceptable mime-type.']} >>> metaweb.upload(" P1,", "text/text", credentials) u'#9202a8c04000641f800000000504e18e' >>> metaweb.upload("P1", "text/text", credentials) u'#9202a8c04000641f800000000504e182' >>> metaweb.upload("P2.foo", "text/text", credentials) {u'status': u'400', u'messages': [u'image/x-portable-graymap is not an acceptable mime-type.']} As you can see, content that begins "P1," causes an error. Some other variants like the last also cause bad behavior. Also, the error JSON that comes back is formatted in a way that breaks the normal upload routine, which expects the 'messages' slot to contain a list of dicts. Chasing this a bit further, it looks like something bad happens (Ajax hang) if you try to make a topic description that begins "P1," in the web interface as well. I guess something is trying to be clever about inferring the content type from the content. Unfortunately, it's being overly clever in this case since I actually do have a text string that begins this way! ("P1,P4-bis(5'-Guanosyl) tetraphosphate is an intermediate in the metabolism of Pyrimidine. It is a substrate for Bis(5'-nucleosyl)-tetraphosphatase.", if you are curious). From jg at metaweb.com Sun Apr 1 01:43:04 2007 From: jg at metaweb.com (John Giannandrea) Date: Sat, 31 Mar 2007 18:43:04 -0700 Subject: [Developers] bug: upload fails for certain strings In-Reply-To: References: Message-ID: Hi We have had some bugs around content type sniffing. You might try setting the mime type to text/plain. Are you uploading to sandbox? if so could you give us your login id or an approx log time so we can find the actual upload entries. Thanks. -jg On Mar 31, 2007, at 5:46 PM, Mike Travers wrote: > [Apologies if this isn't the right place for bug reports.] > > I'm bulk-uploading some data and ran into a bug. The upload service > fails for certain strings. Here's what I see, using the Python upload > routine included in the API documentation: > >>>> metaweb.upload("random string", "text/text", credentials) > u'#9202a8c04000641f800000000504e188' >>>> metaweb.upload("P1,", "text/text", credentials) > {u'status': u'400', u'messages': [u'image/x-portable-bitmap is not an > acceptable mime-type.']} > >>>> metaweb.upload(" P1,", "text/text", credentials) > u'#9202a8c04000641f800000000504e18e' >>>> metaweb.upload("P1", "text/text", credentials) > u'#9202a8c04000641f800000000504e182' >>>> metaweb.upload("P2.foo", "text/text", credentials) > {u'status': u'400', u'messages': [u'image/x-portable-graymap is not an > acceptable mime-type.']} > > As you can see, content that begins "P1," causes an error. Some other > variants like the last also cause bad behavior. Also, the error JSON > that comes back is formatted in a way that breaks the normal upload > routine, which expects the 'messages' slot to contain a list of dicts. > > Chasing this a bit further, it looks like something bad happens (Ajax > hang) if you try to make a topic description that begins "P1," in the > web interface as well. > > I guess something is trying to be clever about inferring the content > type from the content. Unfortunately, it's being overly clever in this > case since I actually do have a text string that begins this way! > ("P1,P4-bis(5'-Guanosyl) tetraphosphate is an intermediate in the > metabolism of Pyrimidine. It is a substrate for > Bis(5'-nucleosyl)-tetraphosphatase.", if you are curious). > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From mt at alum.mit.edu Sun Apr 1 01:54:01 2007 From: mt at alum.mit.edu (Mike Travers) Date: Sat, 31 Mar 2007 18:54:01 -0700 Subject: [Developers] bug: upload fails for certain strings In-Reply-To: References: Message-ID: Yes, this is to sandbox, my login ID is "mt" and I've been doing this sporadically all this afternoon, and again just now. Changing to text/plain doesn't seem to help. Mike On 3/31/07, John Giannandrea wrote: > > Hi > We have had some bugs around content type sniffing. > You might try setting the mime type to text/plain. > Are you uploading to sandbox? if so could you give us > your login id or an approx log time so we can find the > actual upload entries. > Thanks. > -jg > > On Mar 31, 2007, at 5:46 PM, Mike Travers wrote: > > > [Apologies if this isn't the right place for bug reports.] > > > > I'm bulk-uploading some data and ran into a bug. The upload service > > fails for certain strings. Here's what I see, using the Python upload > > routine included in the API documentation: > > > >>>> metaweb.upload("random string", "text/text", credentials) > > u'#9202a8c04000641f800000000504e188' > >>>> metaweb.upload("P1,", "text/text", credentials) > > {u'status': u'400', u'messages': [u'image/x-portable-bitmap is not an > > acceptable mime-type.']} > > > >>>> metaweb.upload(" P1,", "text/text", credentials) > > u'#9202a8c04000641f800000000504e18e' > >>>> metaweb.upload("P1", "text/text", credentials) > > u'#9202a8c04000641f800000000504e182' > >>>> metaweb.upload("P2.foo", "text/text", credentials) > > {u'status': u'400', u'messages': [u'image/x-portable-graymap is not an > > acceptable mime-type.']} > > > > As you can see, content that begins "P1," causes an error. Some other > > variants like the last also cause bad behavior. Also, the error JSON > > that comes back is formatted in a way that breaks the normal upload > > routine, which expects the 'messages' slot to contain a list of dicts. > > > > Chasing this a bit further, it looks like something bad happens (Ajax > > hang) if you try to make a topic description that begins "P1," in the > > web interface as well. > > > > I guess something is trying to be clever about inferring the content > > type from the content. Unfortunately, it's being overly clever in this > > case since I actually do have a text string that begins this way! > > ("P1,P4-bis(5'-Guanosyl) tetraphosphate is an intermediate in the > > metabolism of Pyrimidine. It is a substrate for > > Bis(5'-nucleosyl)-tetraphosphatase.", if you are curious). > > _______________________________________________ > > Developers mailing list > > Developers at freebase.com > > http://lists.freebase.com/mailman/listinfo/developers > > From alecf at metaweb.com Mon Apr 2 21:44:39 2007 From: alecf at metaweb.com (Alec Flett) Date: Mon, 02 Apr 2007 14:44:39 -0700 Subject: [Developers] "index" on any_reverse/any_master Message-ID: <46117947.8070100@metaweb.com> So I've got a query structured like this: { "q":{ "cursor":true, "query":[{ "/type/reflect/any_reverse":{ "id":"#9202a8c04000641f800000000029cf71", "link":{ "master_property":"/film/film/written_by", "timestamp":null } }, "id":null, "limit":30 }] } } (This is "film writers of war of the worlds") The problem is, I really want to sort this query by "index" on the "any_reverse" link. I can't really figure out where to put the "index": null clause. If I put it directly inside any_reverse, or even inside the "link", I get "Must specify index in a list subquery, not a unique subquery" Thoughts? I need to structure my query inside a list because I need the cursor.... Alec From alecf at metaweb.com Mon Apr 2 22:44:37 2007 From: alecf at metaweb.com (Alec Flett) Date: Mon, 02 Apr 2007 15:44:37 -0700 Subject: [Developers] "index" on any_reverse/any_master In-Reply-To: <53815B81-A71D-438C-9342-37CBAFBC1215@metaweb.com> References: <46117947.8070100@metaweb.com> <53815B81-A71D-438C-9342-37CBAFBC1215@metaweb.com> Message-ID: <46118755.9040006@metaweb.com> Unfortunately, I need the actual index that links ...#9202a8c04000641f800000000029cf71 to the outer links along the written_by property... This is what it looks like inverted, doing what I want: {"id": "#9202a8c04000641f800000000029cf71", "/film/film/written_by": [{ "timestamp": null, "index":null, "sort": "index" }] } But I can't invert it because I need a cursor. Alec > > does this do what you want? > > { > "q":{ > "cursor":true, > "query":[{ > "/type/reflect/any_reverse":[{ > "id":"#9202a8c04000641f800000000029cf71", > "index":null, > "link":{ > "master_property":"/film/film/written_by", > "timestamp":null > } > }], > "id":null, > "limit":30, > "sort":"/type/reflect/any_reverse.link.timestamp" > }] > } > } > From tim at metaweb.com Tue Apr 3 00:13:32 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 2 Apr 2007 17:13:32 -0700 Subject: [Developers] Weekly Sandbox refresh Message-ID: Every Monday, usually between 4pm and 6pm Pacific Time (2300-0100 UTC), we erase all data from sandbox.freebase.com and update it with a fresh snapshot from www.freebase.com. This week, we're running a little late; we expect the update to occur sometime in the next two hours. Tim Kientzle Metaweb Technologies, Inc. From niallo at metaweb.com Tue Apr 3 00:38:24 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 02 Apr 2007 17:38:24 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: References: Message-ID: <4611A200.3020504@metaweb.com> The sandbox data refresh has just been completed. Tim Kientzle wrote: > Every Monday, usually between 4pm and 6pm Pacific Time (2300-0100 UTC), > we erase all data from sandbox.freebase.com and update it with a fresh > snapshot from www.freebase.com. > > This week, we're running a little late; we expect the update > to occur sometime in the next two hours. > > Tim Kientzle > Metaweb Technologies, Inc. > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers > -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From william at scissor.com Tue Apr 3 00:57:46 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 17:57:46 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611A200.3020504@metaweb.com> References: <4611A200.3020504@metaweb.com> Message-ID: <4611A68A.8010103@scissor.com> Is the sandbox site otherwise broken? I just get a blank screen when I log in. Thanks, William Niall O'Higgins wrote: > The sandbox data refresh has just been completed. > > Tim Kientzle wrote: > >> Every Monday, usually between 4pm and 6pm Pacific Time (2300-0100 UTC), >> we erase all data from sandbox.freebase.com and update it with a fresh >> snapshot from www.freebase.com. >> >> This week, we're running a little late; we expect the update >> to occur sometime in the next two hours. >> >> Tim Kientzle >> Metaweb Technologies, Inc. >> >> _______________________________________________ >> Developers mailing list >> Developers at freebase.com >> http://lists.freebase.com/mailman/listinfo/developers >> >> > > > From jg at metaweb.com Tue Apr 3 00:58:47 2007 From: jg at metaweb.com (John Giannandrea) Date: Mon, 2 Apr 2007 17:58:47 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611A68A.8010103@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> Message-ID: <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> works for me. -jg On Apr 2, 2007, at 5:57 PM, William Pietri wrote: > Is the sandbox site otherwise broken? I just get a blank screen when I > log in. From william at scissor.com Tue Apr 3 01:13:10 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 18:13:10 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> Message-ID: <4611AA26.1050801@scissor.com> Interesting. All the dynamic stuff is still broken for me. The only pages that work are static ones: http://sandbox.freebase.com/view/tutorial/ http://sandbox.freebase.com/view/showcase And on the home page, I get the sidebar, but nothing else. This happens for me in Firefox 1.5, even after restarting it. It also is what I get in IE 6, if that helps. William John Giannandrea wrote: > > works for me. > -jg > > On Apr 2, 2007, at 5:57 PM, William Pietri wrote: >> Is the sandbox site otherwise broken? I just get a blank screen when I >> log in. > From colin at metaweb.com Tue Apr 3 01:17:01 2007 From: colin at metaweb.com (Colin Evans) Date: Mon, 02 Apr 2007 18:17:01 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AA26.1050801@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> Message-ID: <4611AB0D.5000909@metaweb.com> Login is fubar as well - 'Error contacting server.' William Pietri wrote: > Interesting. All the dynamic stuff is still broken for me. The only > pages that work are static ones: > > http://sandbox.freebase.com/view/tutorial/ > http://sandbox.freebase.com/view/showcase > > And on the home page, I get the sidebar, but nothing else. > > This happens for me in Firefox 1.5, even after restarting it. It also is > what I get in IE 6, if that helps. > > William > > > John Giannandrea wrote: > >> works for me. >> -jg >> >> On Apr 2, 2007, at 5:57 PM, William Pietri wrote: >> >>> Is the sandbox site otherwise broken? I just get a blank screen when I >>> log in. >>> > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070402/1f2b5f10/attachment.htm From faye at metaweb.com Tue Apr 3 01:19:25 2007 From: faye at metaweb.com (Faye Li) Date: Mon, 02 Apr 2007 18:19:25 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AA26.1050801@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> Message-ID: <4611AB9D.2020902@metaweb.com> Hi William, I think you're no longer logged in, and queries that need authentication (i.e. anything but the static content) are all failing. Try login again at: http://sandbox.freebase.com/signin/signin -- Faye William Pietri wrote: > Interesting. All the dynamic stuff is still broken for me. The only > pages that work are static ones: > > http://sandbox.freebase.com/view/tutorial/ > http://sandbox.freebase.com/view/showcase > > And on the home page, I get the sidebar, but nothing else. > > This happens for me in Firefox 1.5, even after restarting it. It also is > what I get in IE 6, if that helps. > > William > > > John Giannandrea wrote: > >> works for me. >> -jg >> >> On Apr 2, 2007, at 5:57 PM, William Pietri wrote: >> >>> Is the sandbox site otherwise broken? I just get a blank screen when I >>> log in. >>> > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070402/2ec9cbb8/attachment.htm From william at scissor.com Tue Apr 3 01:23:10 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 18:23:10 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AB9D.2020902@metaweb.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> <4611AB9D.2020902@metaweb.com> Message-ID: <4611AC7E.5050500@scissor.com> Faye Li wrote: > Hi William, > > I think you're no longer logged in, and queries that need > authentication (i.e. anything but the static content) are all failing. > Try login again at: > > http://sandbox.freebase.com/signin/signin Logging in gets me the same result, that or bounced back to the signin screen. Clearing cookies and clearing cache have not helped appreciably either. I just discovered, though, that this happens only for one of my accounts. The account "querybuilder_bot" doesn't work, but "william" does. William From tim at metaweb.com Tue Apr 3 01:45:22 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 2 Apr 2007 18:45:22 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AC7E.5050500@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> <4611AB9D.2020902@metaweb.com> <4611AC7E.5050500@scissor.com> Message-ID: This problem has been identified and fixed. When we copied the graph data over, the separate authentication database (which stores passwords and other private account data) did not get copied over. All account data from www.freebase.com has been copied over to sandbox.freebase.com, so accounts created in the last week on www.freebase.com now work correctly on sandbox.freebase.com. Tim Kientzle On Apr 2, 2007, at 6:23 PM, William Pietri wrote: > Faye Li wrote: >> Hi William, >> >> I think you're no longer logged in, and queries that need >> authentication (i.e. anything but the static content) are all >> failing. >> Try login again at: >> >> http://sandbox.freebase.com/signin/signin > > Logging in gets me the same result, that or bounced back to the signin > screen. Clearing cookies and clearing cache have not helped > appreciably > either. > > I just discovered, though, that this happens only for one of my > accounts. The account "querybuilder_bot" doesn't work, but > "william" does. > > William > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From william at scissor.com Tue Apr 3 02:49:42 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 19:49:42 -0700 Subject: [Developers] Query Builder 0.5 Message-ID: <4611C0C6.5060103@scissor.com> Thanks to all for feedback on Query Builder 0.4. Based on that, I bring you Query Builder 0.5: http://dev.scissor.com/querybuilder/ New features: * nested browsing and selection of type properties * filtering support for exact values of o dates o booleans o floating-point numbers o integers * the ability to open a query in the query editor That first one is particularly exciting to people. For example, you can now easily build a search for albums by groups in the same genre as The Police. If I counted right, that's seven clicks and a little typing. Known problems: * just as ugly as before * the type browser sometimes jumps back to the top * the "open on sandbox" button still works even for those not logged in to sandbox As before, suggestions and comments are warmly welcomed! William From vtalwar at berkeley.edu Tue Apr 3 04:33:16 2007 From: vtalwar at berkeley.edu (vtalwar at berkeley.edu) Date: Mon, 2 Apr 2007 21:33:16 -0700 (PDT) Subject: [Developers] mjt xhtml In-Reply-To: References: Message-ID: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> Hi all, The thought of using mjt to create svg's recently popped into my head. I tried making a simple example but a crippling problem arose (in Firefox 2.0) which seems to have to do with XML vs. HTML. I spent about a couple of hours looking through mjt.js for the source to no avail. Essentially what's happening is that when the attributes are generated for "top", you get something like this trying to be created in the XML document: (this markup is from the flatten_markup function) This is fine in the HTML document but is problematic for the XML document since it doesn't allow for duplicate attributes (the id attr). The end result is that setting the innerHTML value for tmpdiv in replace_html() has no effect (since the xml doesn't validate I guess? not sure...). I got around this problem by changing using createElementNS with "http://www.w3.org/TR/REC-html40" to create tempdiv and also the original "top" div but then document.getElementById doesn't seem to be able to find "top" (maybe since it has a different namespace now? not sure again...). Any suggestions would be much appreciated :) Thanks, Vishal From nix at metaweb.com Tue Apr 3 19:48:00 2007 From: nix at metaweb.com (Nick Thompson) Date: Tue, 03 Apr 2007 12:48:00 -0700 Subject: [Developers] mjt xhtml In-Reply-To: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> Message-ID: <4612AF70.9090603@metaweb.com> vtalwar at berkeley.edu wrote: > The thought of using mjt to create svg's recently popped into my head. neat! i'd thought about trying this but figured that xml vs html issues would make it too hard - but it sounds like you already got pretty far. if it's making it as far as the double id="..." attribute then i think the chances are good that it can work, at least in firefox. > Essentially what's happening is that when the attributes are generated for > "top", you get something like this trying to be created in the XML > document: > > there is a mjt.js bug here... as you point out, the browser tolerates the duplicated id="top" in html, but xml is not so forgiving. i'll be pushing out a new version of mjt.js this week with a fix for this bug as well as some other goodies. i need to test the new goodies on ie6 first :-/. nick > > (this markup is from the flatten_markup function) > This is fine in the HTML document but is problematic for the XML document > since it doesn't allow for duplicate attributes (the id attr). The end > result is that setting the innerHTML value for tmpdiv in replace_html() > has no effect (since the xml doesn't validate I guess? not sure...). I got > around this problem by changing using createElementNS with > "http://www.w3.org/TR/REC-html40" to create tempdiv and also the original > "top" div but then document.getElementById doesn't seem to be able to find > "top" (maybe since it has a different namespace now? not sure again...). > > Any suggestions would be much appreciated :) > > Thanks, > Vishal > > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From brendan at metaweb.com Tue Apr 3 20:22:22 2007 From: brendan at metaweb.com (Brendan Neutra) Date: Tue, 03 Apr 2007 13:22:22 -0700 Subject: [Developers] images and cc In-Reply-To: <4612AF70.9090603@metaweb.com> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> <4612AF70.9090603@metaweb.com> Message-ID: <4612B77E.2030102@metaweb.com> Hi all, A question on the data side of things: I'm developing some great architecture data and I've got a script to produce a google earth file from Freebase data (shameless plug: http://www.neutraweb.com/Richard_Neutra.kml) I've contacted some flickr users who have creative commons license photos that are of interest and have gotten really positive response from them. What's the proper way to attribute these kinds of photos? I know there is a "source" property for images/content. There is also the article you can attach to the photo. This also brings up the thorny issue of: if someone builds a 3rd party app, how do we encourage/push them to expose the attribution of the photo in the app? Brendan From jg at metaweb.com Wed Apr 4 00:27:57 2007 From: jg at metaweb.com (John Giannandrea) Date: Tue, 3 Apr 2007 17:27:57 -0700 Subject: [Developers] images and cc In-Reply-To: <4612B77E.2030102@metaweb.com> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> <4612AF70.9090603@metaweb.com> <4612B77E.2030102@metaweb.com> Message-ID: <20070404002757.GB32611@metaweb.com> Brendan Neutra wrote: > What's the proper way to attribute these kinds of photos? For the mass image import going on on sandbox, soon on the main site, I have been using /common/licensed_object properties on the image object. This includes an explicit statement of license. Provenance is much harder to model since oftentimes you dont know the origin. We need to revise the image template to display these items and allow them to be added after a manual image upload. Ill file a bug for this and for finalizing the schema for image and how it relates to topic and document. -jg From brendan at metaweb.com Wed Apr 4 00:44:32 2007 From: brendan at metaweb.com (Brendan Neutra) Date: Tue, 03 Apr 2007 17:44:32 -0700 Subject: [Developers] images and cc In-Reply-To: <20070404002757.GB32611@metaweb.com> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> <4612AF70.9090603@metaweb.com> <4612B77E.2030102@metaweb.com> <20070404002757.GB32611@metaweb.com> Message-ID: <4612F4F0.2090809@metaweb.com> here's some relevant info: http://creativecommons.org/learnmore/faq#How_do_I_properly_attribute_a_Creative_Commons_licensed_work? and my approach to complying: license: cc by-nc photographer: Ken Mccown source: http://www.flickr.com/photos/kenmccown/ notes: * for now I just put this in the image description, but I agree it should be stored in it's own place * It is probably safe to assume that the photographer would want to keep the same name of the piece (though if it's very uninformative one could put that in the alias) though you don't technically have to do this unless they specify it. * Rather then give the exact url source of the image (which I don't think has much chance of being maintained) I chose to specify the users flick home page as a source for what it's worth. I suppose this is a work in progress... John Giannandrea wrote: > Brendan Neutra wrote: > >> What's the proper way to attribute these kinds of photos? >> > > For the mass image import going on on sandbox, soon on the > main site, I have been using /common/licensed_object > properties on the image object. This includes an > explicit statement of license. Provenance is much harder > to model since oftentimes you dont know the origin. > > We need to revise the image template to display these items > and allow them to be added after a manual image upload. > Ill file a bug for this and for finalizing the schema > for image and how it relates to topic and document. > > -jg > From william at scissor.com Wed Apr 4 01:00:40 2007 From: william at scissor.com (William Pietri) Date: Tue, 03 Apr 2007 18:00:40 -0700 Subject: [Developers] Query Builder 0.5 In-Reply-To: <4611C0C6.5060103@scissor.com> References: <4611C0C6.5060103@scissor.com> Message-ID: <4612F8B8.9060301@scissor.com> William Pietri wrote: > Thanks to all for feedback on Query Builder 0.4. Based on that, I bring > you Query Builder 0.5: > > http://dev.scissor.com/querybuilder/ > I've released a new version of this; let's call it 0.51. The main difference is that property browser (the upper middle pane) now does not jump around like a barefoot child on hot pavement. For those wondering, the problem was apparently a a GWT bug: http://code.google.com/p/google-web-toolkit/issues/detail?id=369 William From william at scissor.com Thu Apr 5 01:17:58 2007 From: william at scissor.com (William Pietri) Date: Wed, 04 Apr 2007 18:17:58 -0700 Subject: [Developers] Querybuilder 0.6 Message-ID: <46144E46.7040103@scissor.com> Thanks again for the valuable feedback. I have released yet another version: http://dev.scissor.com/querybuilder/ This version adds range filtering for integers, floating point numbers and dates. I've rushed a little more than I like with this version, as I'll be on vacation for a few weeks starting Friday morning. If you notice issues, please let me know and I'll fix them promptly. Thanks, William From tim at metaweb.com Mon Apr 9 19:35:45 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 9 Apr 2007 12:35:45 -0700 Subject: [Developers] www.freebase.com operating read-only. Message-ID: We're experiencing a problem with www.freebase.com right now. The service has been degraded to read-only access until we can resolve it. Thank you for your patience, Tim Kientzle Metaweb Technologies, Inc. From mt at alum.mit.edu Mon Apr 9 20:02:08 2007 From: mt at alum.mit.edu (Mike Travers) Date: Mon, 9 Apr 2007 13:02:08 -0700 Subject: [Developers] Query worked last night, today it times out Message-ID: I made a simple page with mjt to show the most recently created objects, using this query: [{ "creator":{name: null}, "limit":100, "name":null, "id": null, "sort":"-timestamp", "timestamp":null, "type": [{name: null, id: null}] }] This worked fine until this morning, now it doesn't. It appears to be timing out (when I run it from the query editor page). Removing the "type" clause seems to fix the timeout, but I'd like to be getting that information. Maybe this is a bad query to be doing since it ranges over the whole universe. But it's strange that the behavior changed abruptly. It works OK on sandbox. Mike From tim at metaweb.com Mon Apr 9 20:40:30 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 9 Apr 2007 13:40:30 -0700 Subject: [Developers] www.freebase.com restored to normal operation. In-Reply-To: References: Message-ID: <24964209-67C4-4949-A6D2-7B3E43897911@metaweb.com> We've reset the master database and everything seems to be working again, so we've restored the system to read-write mode. As far as we can tell, no data was lost. If you notice any unusual behavior, please let us know. Thank you again for your patience, Tim Kientzle Metaweb Technologies, Inc. On Apr 9, 2007, at 12:35 PM, Tim Kientzle wrote: > We're experiencing a problem with www.freebase.com right now. > > The service has been degraded to read-only access until we can > resolve it. > > Thank you for your patience, > > Tim Kientzle > Metaweb Technologies, Inc. > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From tim at metaweb.com Mon Apr 9 20:44:01 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 9 Apr 2007 13:44:01 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: References: Message-ID: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> We were having some system problems starting around 11am PDT. Those seem to be resolved now. Please try again and let us know if you're still having problems. Tim Kientzle On Apr 9, 2007, at 1:02 PM, Mike Travers wrote: > I made a simple page with mjt to show the most recently created > objects, using this query: > [{ > "creator":{name: null}, > "limit":100, > "name":null, > "id": null, > "sort":"-timestamp", > "timestamp":null, > "type": [{name: null, id: null}] > }] > > This worked fine until this morning, now it doesn't. It appears to be > timing out (when I run it from the query editor page). Removing the > "type" clause seems to fix the timeout, but I'd like to be getting > that information. > > Maybe this is a bad query to be doing since it ranges over the whole > universe. But it's strange that the behavior changed abruptly. It > works OK on sandbox. > > Mike > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From mt at alum.mit.edu Mon Apr 9 20:59:29 2007 From: mt at alum.mit.edu (Mike Travers) Date: Mon, 9 Apr 2007 13:59:29 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> References: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> Message-ID: I'm still getting the timeouts as before. Mike On 4/9/07, Tim Kientzle wrote: > We were having some system problems starting around 11am PDT. > Those seem to be resolved now. > > Please try again and let us know if you're still having > problems. > > Tim Kientzle > > On Apr 9, 2007, at 1:02 PM, Mike Travers wrote: > > > I made a simple page with mjt to show the most recently created > > objects, using this query: > > [{ > > "creator":{name: null}, > > "limit":100, > > "name":null, > > "id": null, > > "sort":"-timestamp", > > "timestamp":null, > > "type": [{name: null, id: null}] > > }] > > > > This worked fine until this morning, now it doesn't. It appears to be > > timing out (when I run it from the query editor page). Removing the > > "type" clause seems to fix the timeout, but I'd like to be getting > > that information. > > > > Maybe this is a bad query to be doing since it ranges over the whole > > universe. But it's strange that the behavior changed abruptly. It > > works OK on sandbox. > > > > Mike > > _______________________________________________ > > Developers mailing list > > Developers at freebase.com > > http://lists.freebase.com/mailman/listinfo/developers > > From mt at alum.mit.edu Mon Apr 9 21:30:43 2007 From: mt at alum.mit.edu (Mike Travers) Date: Mon, 9 Apr 2007 14:30:43 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: References: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> Message-ID: Well, now it's working, but there was a short period after your reboot where it wasn't. Mysterious. Mike On 4/9/07, Mike Travers wrote: > I'm still getting the timeouts as before. > > Mike > > On 4/9/07, Tim Kientzle wrote: > > We were having some system problems starting around 11am PDT. > > Those seem to be resolved now. > > > > Please try again and let us know if you're still having > > problems. > > > > Tim Kientzle > > > > On Apr 9, 2007, at 1:02 PM, Mike Travers wrote: > > > > > I made a simple page with mjt to show the most recently created > > > objects, using this query: > > > [{ > > > "creator":{name: null}, > > > "limit":100, > > > "name":null, > > > "id": null, > > > "sort":"-timestamp", > > > "timestamp":null, > > > "type": [{name: null, id: null}] > > > }] > > > > > > This worked fine until this morning, now it doesn't. It appears to be > > > timing out (when I run it from the query editor page). Removing the > > > "type" clause seems to fix the timeout, but I'd like to be getting > > > that information. > > > > > > Maybe this is a bad query to be doing since it ranges over the whole > > > universe. But it's strange that the behavior changed abruptly. It > > > works OK on sandbox. > > > > > > Mike > > > _______________________________________________ > > > Developers mailing list > > > Developers at freebase.com > > > http://lists.freebase.com/mailman/listinfo/developers > > > > > From william at scissor.com Mon Apr 9 22:52:17 2007 From: william at scissor.com (William Pietri) Date: Mon, 09 Apr 2007 15:52:17 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: References: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> Message-ID: <461AC3A1.6060306@scissor.com> Mike Travers wrote: > Well, now it's working, but there was a short period after your reboot > where it wasn't. Mysterious. > For what it's worth, I've seen similarly weird behavior with timeouts. I reported a couple, and then put the rest down to alpha stuff that will get fixed eventually. William From niallo at metaweb.com Mon Apr 9 23:49:23 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 09 Apr 2007 16:49:23 -0700 Subject: [Developers] Weekly Sandbox refresh Message-ID: <461AD103.2010805@metaweb.com> Our weekly refresh of the sandbox data will be happening over the next few minutes. -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From niallo at metaweb.com Mon Apr 9 23:58:27 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 09 Apr 2007 16:58:27 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <461AD103.2010805@metaweb.com> References: <461AD103.2010805@metaweb.com> Message-ID: <461AD323.50403@metaweb.com> Data refresh has been completed. Niall O'Higgins wrote: > Our weekly refresh of the sandbox data will be happening over the next > few minutes. > > -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From mjc at cs.washington.edu Wed Apr 11 20:45:12 2007 From: mjc at cs.washington.edu (Michael J Cafarella) Date: Wed, 11 Apr 2007 13:45:12 -0700 (PDT) Subject: [Developers] MQL and Metaweb data model Message-ID: Hi all, I've been reading through the MQL manual and am trying to figure out how MQL relates to other database query languages. I think I've figured a few things out, but I'd like to make sure I've got it right. A think Metaweb employees will be best-placed to answer a lot of this stuff, but it would be great to hear from anyone who has an opinion. Thanks, --Mike Cafarella ----------------------------------------------------------------- 1) It looks like there is no such thing as a "join" in the sense of creating a brand-new tuple as in a relational system. The only thing that can be returned by a query is a set of objects that already exist in the Metaweb store (or subparts of those objects). 2) However, it is very possible to perform semijoin selections. It's not hard to write a query that returns "all musical artists who are also movie stars". In fact, there seems to be no limit on the width of this semijoin. So it's possible to write a query that expresses: "Show tracks from all albums where the artist appears in a movie that was directed by someone born in city in a state with fewer than 1,000,000 residents" This query expresses a semijoin using the following sets: a) The set of tracks b) The set of albums c) The set of artists d) The set of movies e) The set of people (directors) f) The set of cities g) The set of states I imagine a query like this could be pretty burdensome to compute. 3) Are there implementation-specific limits on resource consumption, either on the number of objects returned or the size of the join? I know it defaults to a limit of 100, but there are no language-level limits in the docs that I see. 4) I can express a semijoin between any two arbitrary sets. Often these sets are all the instances of a given type, but they don't have to be. I think I could express a semijoin between albums and all objects that have "Police" in the name. Unlike the set of instances of a given type, this kind of set has not been materialized before I pose my query. Something like this also seems hard to compute. Do you expect the system to be able to handle this type of query at large scale? 5) It seems like XQuery could work fairly well on the Metaweb data model. Is there a reason you decided against it? Thanks again, --Mike From sm at metaweb.com Thu Apr 12 17:20:34 2007 From: sm at metaweb.com (Scott Meyer) Date: Thu, 12 Apr 2007 10:20:34 -0700 Subject: [Developers] MQL and Metaweb data model Message-ID: <461E6A62.8020503@metaweb.com> > From: Michael J Cafarella > 1) It looks like there is no such thing as a "join" in the sense of > creating a brand-new tuple as in a relational system. The only thing that > can be returned by a query is a set of objects that already exist in the > Metaweb store (or subparts of those objects). Unlike a relational database which deals only with values, identity is fundamental to Freebase. We work with sets of identities, not sets of values. For simplicity, MQL presents results as a nested series of dictionaries which, at a given level, look like semijoin results, but denormalizing these results in the obvious way can yield the table you'd expect from a natural join. There's actually no such thing as a semijoin either, the results just happen to look that way. :-) > 2) However, it is very possible to perform semijoin selections. It's not > hard to write a query that returns "all musical artists who are also movie > stars". In fact, there seems to be no limit on the width of this > semijoin. So it's possible to write a query that expresses: > > "Show tracks from all albums where the artist appears in a movie that was > directed by someone born in city in a state with fewer than 1,000,000 > residents" > > This query expresses a semijoin using the following sets: > > a) The set of tracks > b) The set of albums > c) The set of artists > d) The set of movies > e) The set of people (directors) > f) The set of cities > g) The set of states > > I imagine a query like this could be pretty burdensome to compute. Indeed. If we were doing traditional relational semijoins this would become intractable very quickly. We work directly with sorted sets of identities, which, like 4-wheel drive, allows us to get stuck further from civilization. If you push far enough, things do become computationally intractable, but the current system does surprisingly well on a variety of useful queries. And, we have only just begun to optimize. > 3) Are there implementation-specific limits on resource consumption, > either on the number of objects returned or the size of the join? I know > it defaults to a limit of 100, but there are no language-level limits in > the docs that I see. We limit queries to 8 seconds. There's currently no limit on memory consumption, but we do give up on queries that are "too hard." Note that there is cursor support for returning large datasets a page at a time. > 4) I can express a semijoin between any two arbitrary sets. Often these > sets are all the instances of a given type, but they don't have to be. I > think I could express a semijoin between albums and all objects that have > "Police" in the name. Unlike the set of instances of a given type, this > kind of set has not been materialized before I pose my query. > > Something like this also seems hard to compute. Do you expect the > system to be able to handle this type of query at large scale? Yes. Aside from their identity, the underlying tuple store is completely ignorant of "type" so the "set of instances of a given type" is no different from the "set of all instances with 'Police' in the name" The fact that relational stores conflate 'type' with physical storage - a sore point with Codd, I believe - makes it exceedingly difficult to use a relational store to do what Freebase does. This is one of the reasons that we built our own tuple store. > 5) It seems like XQuery could work fairly well on the Metaweb data model. > Is there a reason you decided against it? If the adoption of XPath or any other particular query syntax were critical to doing what Freebase does, we would have adopted it. We're exploring a new way to store and retrieve data, and the cost of implementing a standard syntax (or the inevitable super/sub/side-set of a standard syntax) far outweighs the benefits. As things evolve, we're completely open to supporting other syntaxes, or helping third parties to do so. The objective of MQL was to present a simple, stable subset of the underlying graph query functionality in a way that was palatable to both programmers and programming languages such as Javscript and Python. Regards, -Scott From mjc at cs.washington.edu Thu Apr 12 18:41:37 2007 From: mjc at cs.washington.edu (Michael J Cafarella) Date: Thu, 12 Apr 2007 11:41:37 -0700 (PDT) Subject: [Developers] MQL and Metaweb data model In-Reply-To: <461E6A62.8020503@metaweb.com> References: <461E6A62.8020503@metaweb.com> Message-ID: Hi Scott, Thanks for your detailed reply. I really appreciate the time you took. It sounds like your database is actually more similar to a search engine's document-store than to a relational database. In a search engine, documents are "identities" which have a bag of associated values. You could imagine marking a document with text that says "band=The_Police" or "type=Album". You could retrieve results for either the "band" query or the "type" query easily. The "wide-semijoin" that I described would entail looking up a number of these different sets, sorting them, then performing an intersection-style operation between them. Like you say, this would be difficult eventually (say, 1000 different types) but you might not run into problems for any real-world queries. This is very interesting stuff. I hope you consider publishing an article about it someday. --Mike From alecf at metaweb.com Fri Apr 13 19:04:31 2007 From: alecf at metaweb.com (Alec Flett) Date: Fri, 13 Apr 2007 12:04:31 -0700 Subject: [Developers] Flash Data Mobs Message-ID: <461FD43F.10404@metaweb.com> Internally at Metaweb, we've been running an almost-weekly "flash data mob" activity. The idea has been to take 5-10 minutes (or more, if you've got the inclination) and fill in some missing data in Freebase. The idea is for a bunch of people to all focus on flushing out one corner of data - the combined power of a bunch of people all doing a little work is really rather impressive. For an example of one of our first Flash Data Mobs, check out Kevin Bacon: http://www.freebase.com/view?id=/wikipedia/en/Kevin_Bacon His entry is very well filled out - not only as an actor but as a person, a director, even a reader of audio books! (Feel free to add more, but keep reading!) A lot of great things come out of this: 1) Unusual issues with the user interfaces - sometimes bugs, and sometimes real problems with the way we present and edit the data. 2) Data modelling issues - there was lots of discussion about what it meant for Kevin to play "himself" on Will and Grace some time back 3) Data source questions - IMDB is great for getting movie/tv data, but what about music? books? biographical information? You get the idea and of course 4) even more quality data in Freebase. This is great for developers because we learn about the data and get inspiration for new mashups. So I'm going to be running a weekly Flash Data Mob - each week we'll have some interesting task or topic that we'll all try to fill out, and we'll take up the discussion on this list. Later today I'll send out our first public Flash Data Mob activity that should give you something fun to do for a lazy friday afternoon.. stay tuned! I'll also be posting this and the first flash mob on our blog at http://blog.freebase.com/ Alec From alecf at metaweb.com Sat Apr 14 02:37:37 2007 From: alecf at metaweb.com (Alec Flett) Date: Fri, 13 Apr 2007 19:37:37 -0700 Subject: [Developers] Friday the 13th Data Flash Mob Message-ID: <46203E71.7050403@metaweb.com> Did I say "Lazy Friday" - I should have said "Friday the 13th" - last minute stuff delayed this a bit, sorry about that. Anyway, here's our first public Data Flash Mob. If you missed my earlier message, see the blog post: http://blog.freebase.com/?p=20 And in the spirit of Friday the 13th, we're going to fill out the Friday the 13th Films! Start here: http://www.freebase.com/view?id=%239202a8c04000641f8000000000134e8e If we all just take a few minutes to fill out the original film and the outgoing links, we can probably connect all of its sequels together pretty easily - not just by film series but also by actors/etc that appeared in the various films. I read recently that New Line Cinema recently acquired rights to this series, so it would be interesting to see if we can do a kevin-bacon-game style link between the original movie and New Line.. see what their connection is/was before they produce the next film. So the goal here is to fill out the Friday the 13th series of films to get the shortest link between New Line - i.e. maybe the directors of one of the sequels directed other movies by New Line? Who knows! New Line seems to be incorrectly typed as Film, so we should fix that too http://www.freebase.com/view?id=%239202a8c04000641f8000000000225dd9 Good luck! Please report back to the list with any feedback, questions, comments, etc... Alec From jonathanicity at gmail.com Sun Apr 15 00:06:15 2007 From: jonathanicity at gmail.com (Siddharth Jonathan) Date: Sat, 14 Apr 2007 17:06:15 -0700 Subject: [Developers] freebase data download Message-ID: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> Hi, I'm new to freebase and I'm still looking around. Can someone tell me what the quickest way is to download all the freebase data? I would like to do that to analyze some of the data. If you can point me to any scripts/information that would be very helpful. Thanks! Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070414/9314fb72/attachment.htm From jg at metaweb.com Sun Apr 15 02:30:59 2007 From: jg at metaweb.com (John Giannandrea) Date: Sat, 14 Apr 2007 19:30:59 -0700 Subject: [Developers] freebase data download In-Reply-To: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> References: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> Message-ID: <370F950D-C553-424B-AB9F-8F6E2AEABB18@metaweb.com> Hi The easiest way to do it is by type, asking for all the instances of a specific type and whatever properties you are interested in. You need to use the cursor feature of MQL to make very large downloads. What language do you want to do this in, we may have something that we can provide that already does this. If you want to download absolutely everything in some specific format, contact me off list and we can discuss the options. -jg On Apr 14, 2007, at 5:06 PM, Siddharth Jonathan wrote: > Hi, > I'm new to freebase and I'm still looking around. > Can someone tell me what the quickest way is to download all the > freebase data? > I would like to do that to analyze some of the data. > If you can point me to any scripts/information that > would be very helpful. > Thanks! > > Jonathan > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From jonathanicity at gmail.com Sun Apr 15 02:33:01 2007 From: jonathanicity at gmail.com (Siddharth Jonathan) Date: Sat, 14 Apr 2007 19:33:01 -0700 Subject: [Developers] freebase data download In-Reply-To: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> References: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> Message-ID: <616305cc0704141933k7d0d4f22rf6d74742a706c008@mail.gmail.com> I guess one simple way appears to be to get all the guids out in pass, and in a second pass, pull out every property for each object referenced by each guid using a wildcard. Jonathan On 4/14/07, Siddharth Jonathan wrote: > > Hi, > I'm new to freebase and I'm still looking around. > Can someone tell me what the quickest way is to download all the freebase > data? > I would like to do that to analyze some of the data. > If you can point me to any scripts/information that would be > very helpful. > Thanks! > > Jonathan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070414/2e956590/attachment.htm From niallo at metaweb.com Mon Apr 16 23:54:24 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 16 Apr 2007 16:54:24 -0700 Subject: [Developers] Weekly sandbox.freebase.com data refresh Message-ID: <46240CB0.1020104@metaweb.com> Hi, The weekly data refresh & software update of sandbox.freebase.com will be starting shortly. -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From niallo at metaweb.com Tue Apr 17 00:52:05 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 16 Apr 2007 17:52:05 -0700 Subject: [Developers] Weekly sandbox.freebase.com data refresh In-Reply-To: <46240CB0.1020104@metaweb.com> References: <46240CB0.1020104@metaweb.com> Message-ID: <46241A35.9040508@metaweb.com> Completed. Niall O'Higgins wrote: > Hi, > > The weekly data refresh & software update of sandbox.freebase.com will > be starting shortly. > > -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From jack at metaweb.com Thu Apr 19 23:59:12 2007 From: jack at metaweb.com (Jack Alves) Date: Thu, 19 Apr 2007 16:59:12 -0700 Subject: [Developers] date constraint by month-day only Message-ID: What is the best way to get topics with a date that includes a specific month and day? I was given the query below to get all topics with a specific full date. MQL docs specify numeric constraint operators for dates. So the only way I can think of to get anything that happened on any Nov 22 is to repeat the query for every year. Is there a more efficient way? Will MQL eventually support a date contraint like "*-11-22"? Here's the query I'm using, { "query":[{ "/type/reflect/any_value":[{ "link":{ "master_property":{ "name":null } }, "type":"/type/datetime", "value":"1963-11-22" }], "id":null, "name":null, "sort":"/type/reflect/any_value.link.master_property.name", "type":"/common/topic" }] } Here's a link to a mjt script that uses the above query, http://dev.mqlx.com/%7Ejack/mjt/dayHistory.html?year=1963&day=11-22 Here's a link to the queryeditor with the above query, http://www.freebase.com/view/queryeditor/?q={%20%22query%22:[{%20%22/type/re flect/any_value%22:[{%20%22link%22:{%20%22master_property%22:{%20%22name%22: null%20}%20},%20%22type%22:%22/type/datetime%22,%20%22value%22:%221963-11-22 %22%20}],%20%22id%22:null,%20%22name%22:null,%20%22sort%22:%22/type/reflect/ any_value.link.master_property.name%22,%20%22type%22:%22/common/topic%22%20} ]%20%7d From jg at metaweb.com Fri Apr 20 01:45:19 2007 From: jg at metaweb.com (John Giannandrea) Date: Thu, 19 Apr 2007 18:45:19 -0700 Subject: [Developers] date constraint by month-day only In-Reply-To: References: Message-ID: <93F6AFAF-4D14-4AA9-9900-70F6D09113EC@metaweb.com> You cant currently use ~= with datetime. It used to work, we should reconsider it. -jg Jack Alves wrote: > What is the best way to get topics with a date that includes a > specific > month and day? I was given the query below to get all topics with a > specific > full date. MQL docs specify numeric constraint operators for dates. > So the > only way I can think of to get anything that happened on any Nov 22 > is to > repeat the query for every year. Is there a more efficient way? > Will MQL > eventually support a date contraint like "*-11-22"? From alecf at metaweb.com Mon Apr 23 17:23:02 2007 From: alecf at metaweb.com (Alec Flett) Date: Mon, 23 Apr 2007 10:23:02 -0700 Subject: [Developers] Flash Data Mob #2: Spinoffs Message-ID: <462CEB76.7090309@metaweb.com> I found this really cool page on Wikipedia: http://en.wikipedia.org/wiki/List_of_television_spin-offs This is a whole list of all the shows that have spun off of other shows. Many people have been painstakingly editing this page to keep this accurate and in sync with their respective articles, but this is an area where Freebase really shines - once the data is in there, it's entirely possible to render this page programmatically. So the challenge here is a two parter - one for the TV geeks to enter as much of this data as possible, and one for some MJT hackers. Here are the details: 1) Look through this list and find at least 5 shows with spin-offs, and connect them up in freebase. Just search for their titles in the search box that's on every freebase page. I've already hooked up the Happy Days chain (who knew Happy Days itself was a spin-off?) http://www.freebase.com/view?id=%239202a8c04000641f80000000001379b3 To get you started, here are a few of the more interesting "roots" of these spinoff trees.. you'll find some real gems even if you don't recognize the root show... Looney Toons: http://www.freebase.com/view?id=%239202a8c04000641f8000000000062641 Sam and Friends: http://www.freebase.com/view?id=%239202a8c04000641f80000000003fccf6 Broadway Open House: http://www.freebase.com/view?id=%239202a8c04000641f800000000019716d Cheers: http://www.freebase.com/view?id=%239202a8c04000641f800000000000e101 Diff'ren Strokes: http://www.freebase.com/view?id=%239202a8c04000641f800000000005bbc2 Law & Order: http://www.freebase.com/view?id=%239202a8c04000641f800000000013b473 (and yeah, we're fixing that '&' problem!) All in the Family: http://www.freebase.com/view?id=%239202a8c04000641f800000000012d68a 2) The MJT challenge see if you can render the equivalent of the Wikipedia page with MJT. This is a good test of MQL and MJT - how easy is it to query AND render this multi-level hierarchy? This is potentially a weak part of MQL, but I'd love to see how some creative folks approach the problem. Alec From tsturge at metaweb.com Mon Apr 23 17:40:43 2007 From: tsturge at metaweb.com (Tim Sturge) Date: Mon, 23 Apr 2007 10:40:43 -0700 Subject: [Developers] date constraint by month-day only In-Reply-To: References: Message-ID: <462CEF9B.30009@metaweb.com> This is a hard problem with MQL as it currently stands. There's no good solution right now beyond asking for everything and filtering or asking for each year in a separate query; neither work very well. I quite like the syntax "*-11-22" but it would need a fair bit of work internally (firstly so as not to be horribly inefficient, and secondly so as not to match 11:22 AM as well -- the current ~= syntax implies that - and : are equivalent) I'm going to take this and add it to the MQL requested features list. (which doesn't come with any particular guarantee of action unfortunately :-) ) One feature that will probably arrive sooner will be the ability to use |= lists on values, so soon you will be able to say: "date": { "value|=": ["2006-11-22","2005-11-22","2004-11-22", ... ], "value": null } This doesn't relieve you of having to spell out each year explicitly, but it will perform quite well for <= 100 values (a full century of Nov 22s) and may be sufficient in some cases (hard to say from the query below what the desired results are going to look like.) Tim Jack Alves wrote: > What is the best way to get topics with a date that includes a specific > month and day? I was given the query below to get all topics with a specific > full date. MQL docs specify numeric constraint operators for dates. So the > only way I can think of to get anything that happened on any Nov 22 is to > repeat the query for every year. Is there a more efficient way? Will MQL > eventually support a date contraint like "*-11-22"? > > Here's the query I'm using, > { > "query":[{ > "/type/reflect/any_value":[{ > "link":{ > "master_property":{ > "name":null > } > }, > "type":"/type/datetime", > "value":"1963-11-22" > }], > "id":null, > "name":null, > "sort":"/type/reflect/any_value.link.master_property.name", > "type":"/common/topic" > }] > } > > Here's a link to a mjt script that uses the above query, > http://dev.mqlx.com/%7Ejack/mjt/dayHistory.html?year=1963&day=11-22 > > > Here's a link to the queryeditor with the above query, > http://www.freebase.com/view/queryeditor/?q={%20%22query%22:[{%20%22/type/re > flect/any_value%22:[{%20%22link%22:{%20%22master_property%22:{%20%22name%22: > null%20}%20},%20%22type%22:%22/type/datetime%22,%20%22value%22:%221963-11-22 > %22%20}],%20%22id%22:null,%20%22name%22:null,%20%22sort%22:%22/type/reflect/ > any_value.link.master_property.name%22,%20%22type%22:%22/common/topic%22%20} > ]%20%7d > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers > From niallo at metaweb.com Tue Apr 24 00:17:14 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 23 Apr 2007 17:17:14 -0700 Subject: [Developers] Sandbox.freebase.com data refresh and software update Message-ID: <462D4C8A.8070805@metaweb.com> Kicking off the data refresh and software update now. -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From niallo at metaweb.com Tue Apr 24 00:43:13 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 23 Apr 2007 17:43:13 -0700 Subject: [Developers] Sandbox.freebase.com data refresh and software update In-Reply-To: <462D4C8A.8070805@metaweb.com> References: <462D4C8A.8070805@metaweb.com> Message-ID: <462D52A1.70805@metaweb.com> Update complete. Niall O'Higgins wrote: > Kicking off the data refresh and software update now. > > -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From alecf at metaweb.com Wed Apr 25 15:36:02 2007 From: alecf at metaweb.com (Alec Flett) Date: Wed, 25 Apr 2007 08:36:02 -0700 Subject: [Developers] Flash Data Mob #2: Spinoffs In-Reply-To: <462CEB76.7090309@metaweb.com> References: <462CEB76.7090309@metaweb.com> Message-ID: <462F7562.8060904@metaweb.com> Wow, this is definitely the most successful Flash Data Mob to date... all the work of two folks: 1) Colin Evans, who wrote a script to scrape the wikipedia page - we now have probably-complete spin-off information for all the TV shows Freebase knows about. 2) Will Moffat who wrote a MJT script and mimicked the wikipedia page almost exactly: http://hamstersoup.wordpress.com/2007/04/24/playing-with-freebase-list-of-television-spin-offs/ - this is a great MJT example too and I'll see what we can do to feature it on the Freebase developer page. Nice job guys...(and thanks to Dae Park for finding Will's blog entry!) Alec Alec Flett wrote: > I found this really cool page on Wikipedia: > http://en.wikipedia.org/wiki/List_of_television_spin-offs > > This is a whole list of all the shows that have spun off of other shows. > > Many people have been painstakingly editing this page to keep this > accurate and in sync with their respective articles, but this is an > area where Freebase really shines - once the data is in there, it's > entirely possible to render this page programmatically. > > So the challenge here is a two parter - one for the TV geeks to enter > as much of this data as possible, and one for some MJT hackers. > > Here are the details: > 1) Look through this list and find at least 5 shows with spin-offs, > and connect them up in freebase. Just search for their titles in the > search box that's on every freebase page. I've already hooked up the > Happy Days chain (who knew Happy Days itself was a spin-off?) > http://www.freebase.com/view?id=%239202a8c04000641f80000000001379b3 > > To get you started, here are a few of the more interesting "roots" of > these spinoff trees.. you'll find some real gems even if you don't > recognize the root show... > > Looney Toons: > http://www.freebase.com/view?id=%239202a8c04000641f8000000000062641 > Sam and Friends: > http://www.freebase.com/view?id=%239202a8c04000641f80000000003fccf6 > Broadway Open House: > http://www.freebase.com/view?id=%239202a8c04000641f800000000019716d > Cheers: > http://www.freebase.com/view?id=%239202a8c04000641f800000000000e101 > Diff'ren Strokes: > http://www.freebase.com/view?id=%239202a8c04000641f800000000005bbc2 > Law & Order: > http://www.freebase.com/view?id=%239202a8c04000641f800000000013b473 > (and yeah, we're fixing that '&' problem!) > All in the Family: > http://www.freebase.com/view?id=%239202a8c04000641f800000000012d68a > > > 2) The MJT challenge see if you can render the equivalent of the > Wikipedia page with MJT. This is a good test of MQL and MJT - how easy > is it to query AND render this multi-level hierarchy? This is > potentially a weak part of MQL, but I'd love to see how some creative > folks approach the problem. > > Alec > > > From alecf at metaweb.com Thu Apr 26 20:03:23 2007 From: alecf at metaweb.com (Alec Flett) Date: Thu, 26 Apr 2007 13:03:23 -0700 Subject: [Developers] changing the apis for search/autocomplete Message-ID: <4631058B.2000405@metaweb.com> If you're using the search and/or autocomplete JSON APIs, can you please drop me an e-mail? In the next few weeks we're going to make some minor tweaks to these APIs. We don't know exactly what the changes are, but they'll probably be reasonably minor including standardizing success/error messages, slight changes to the names of the fields, and slight changes to the names of the form parameters. I'll post more details when we know them, but I wanted to give any developers a heads up. (And if you're not using them, good :) We do intend for them to be generally consumed, but now is probably not the time to start looking into them) Alec From aseem at metaweb.com Fri Apr 13 00:18:47 2007 From: aseem at metaweb.com (Aseem Mohanty) Date: Fri, 13 Apr 2007 00:18:47 -0000 Subject: [Developers] subscribe Message-ID: <461ECC5D.1060609@metaweb.com> From mt at alum.mit.edu Sun Apr 1 00:46:19 2007 From: mt at alum.mit.edu (Mike Travers) Date: Sat, 31 Mar 2007 17:46:19 -0700 Subject: [Developers] bug: upload fails for certain strings Message-ID: [Apologies if this isn't the right place for bug reports.] I'm bulk-uploading some data and ran into a bug. The upload service fails for certain strings. Here's what I see, using the Python upload routine included in the API documentation: >>> metaweb.upload("random string", "text/text", credentials) u'#9202a8c04000641f800000000504e188' >>> metaweb.upload("P1,", "text/text", credentials) {u'status': u'400', u'messages': [u'image/x-portable-bitmap is not an acceptable mime-type.']} >>> metaweb.upload(" P1,", "text/text", credentials) u'#9202a8c04000641f800000000504e18e' >>> metaweb.upload("P1", "text/text", credentials) u'#9202a8c04000641f800000000504e182' >>> metaweb.upload("P2.foo", "text/text", credentials) {u'status': u'400', u'messages': [u'image/x-portable-graymap is not an acceptable mime-type.']} As you can see, content that begins "P1," causes an error. Some other variants like the last also cause bad behavior. Also, the error JSON that comes back is formatted in a way that breaks the normal upload routine, which expects the 'messages' slot to contain a list of dicts. Chasing this a bit further, it looks like something bad happens (Ajax hang) if you try to make a topic description that begins "P1," in the web interface as well. I guess something is trying to be clever about inferring the content type from the content. Unfortunately, it's being overly clever in this case since I actually do have a text string that begins this way! ("P1,P4-bis(5'-Guanosyl) tetraphosphate is an intermediate in the metabolism of Pyrimidine. It is a substrate for Bis(5'-nucleosyl)-tetraphosphatase.", if you are curious). From jg at metaweb.com Sun Apr 1 01:43:04 2007 From: jg at metaweb.com (John Giannandrea) Date: Sat, 31 Mar 2007 18:43:04 -0700 Subject: [Developers] bug: upload fails for certain strings In-Reply-To: References: Message-ID: Hi We have had some bugs around content type sniffing. You might try setting the mime type to text/plain. Are you uploading to sandbox? if so could you give us your login id or an approx log time so we can find the actual upload entries. Thanks. -jg On Mar 31, 2007, at 5:46 PM, Mike Travers wrote: > [Apologies if this isn't the right place for bug reports.] > > I'm bulk-uploading some data and ran into a bug. The upload service > fails for certain strings. Here's what I see, using the Python upload > routine included in the API documentation: > >>>> metaweb.upload("random string", "text/text", credentials) > u'#9202a8c04000641f800000000504e188' >>>> metaweb.upload("P1,", "text/text", credentials) > {u'status': u'400', u'messages': [u'image/x-portable-bitmap is not an > acceptable mime-type.']} > >>>> metaweb.upload(" P1,", "text/text", credentials) > u'#9202a8c04000641f800000000504e18e' >>>> metaweb.upload("P1", "text/text", credentials) > u'#9202a8c04000641f800000000504e182' >>>> metaweb.upload("P2.foo", "text/text", credentials) > {u'status': u'400', u'messages': [u'image/x-portable-graymap is not an > acceptable mime-type.']} > > As you can see, content that begins "P1," causes an error. Some other > variants like the last also cause bad behavior. Also, the error JSON > that comes back is formatted in a way that breaks the normal upload > routine, which expects the 'messages' slot to contain a list of dicts. > > Chasing this a bit further, it looks like something bad happens (Ajax > hang) if you try to make a topic description that begins "P1," in the > web interface as well. > > I guess something is trying to be clever about inferring the content > type from the content. Unfortunately, it's being overly clever in this > case since I actually do have a text string that begins this way! > ("P1,P4-bis(5'-Guanosyl) tetraphosphate is an intermediate in the > metabolism of Pyrimidine. It is a substrate for > Bis(5'-nucleosyl)-tetraphosphatase.", if you are curious). > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From mt at alum.mit.edu Sun Apr 1 01:54:01 2007 From: mt at alum.mit.edu (Mike Travers) Date: Sat, 31 Mar 2007 18:54:01 -0700 Subject: [Developers] bug: upload fails for certain strings In-Reply-To: References: Message-ID: Yes, this is to sandbox, my login ID is "mt" and I've been doing this sporadically all this afternoon, and again just now. Changing to text/plain doesn't seem to help. Mike On 3/31/07, John Giannandrea wrote: > > Hi > We have had some bugs around content type sniffing. > You might try setting the mime type to text/plain. > Are you uploading to sandbox? if so could you give us > your login id or an approx log time so we can find the > actual upload entries. > Thanks. > -jg > > On Mar 31, 2007, at 5:46 PM, Mike Travers wrote: > > > [Apologies if this isn't the right place for bug reports.] > > > > I'm bulk-uploading some data and ran into a bug. The upload service > > fails for certain strings. Here's what I see, using the Python upload > > routine included in the API documentation: > > > >>>> metaweb.upload("random string", "text/text", credentials) > > u'#9202a8c04000641f800000000504e188' > >>>> metaweb.upload("P1,", "text/text", credentials) > > {u'status': u'400', u'messages': [u'image/x-portable-bitmap is not an > > acceptable mime-type.']} > > > >>>> metaweb.upload(" P1,", "text/text", credentials) > > u'#9202a8c04000641f800000000504e18e' > >>>> metaweb.upload("P1", "text/text", credentials) > > u'#9202a8c04000641f800000000504e182' > >>>> metaweb.upload("P2.foo", "text/text", credentials) > > {u'status': u'400', u'messages': [u'image/x-portable-graymap is not an > > acceptable mime-type.']} > > > > As you can see, content that begins "P1," causes an error. Some other > > variants like the last also cause bad behavior. Also, the error JSON > > that comes back is formatted in a way that breaks the normal upload > > routine, which expects the 'messages' slot to contain a list of dicts. > > > > Chasing this a bit further, it looks like something bad happens (Ajax > > hang) if you try to make a topic description that begins "P1," in the > > web interface as well. > > > > I guess something is trying to be clever about inferring the content > > type from the content. Unfortunately, it's being overly clever in this > > case since I actually do have a text string that begins this way! > > ("P1,P4-bis(5'-Guanosyl) tetraphosphate is an intermediate in the > > metabolism of Pyrimidine. It is a substrate for > > Bis(5'-nucleosyl)-tetraphosphatase.", if you are curious). > > _______________________________________________ > > Developers mailing list > > Developers at freebase.com > > http://lists.freebase.com/mailman/listinfo/developers > > From alecf at metaweb.com Mon Apr 2 21:44:39 2007 From: alecf at metaweb.com (Alec Flett) Date: Mon, 02 Apr 2007 14:44:39 -0700 Subject: [Developers] "index" on any_reverse/any_master Message-ID: <46117947.8070100@metaweb.com> So I've got a query structured like this: { "q":{ "cursor":true, "query":[{ "/type/reflect/any_reverse":{ "id":"#9202a8c04000641f800000000029cf71", "link":{ "master_property":"/film/film/written_by", "timestamp":null } }, "id":null, "limit":30 }] } } (This is "film writers of war of the worlds") The problem is, I really want to sort this query by "index" on the "any_reverse" link. I can't really figure out where to put the "index": null clause. If I put it directly inside any_reverse, or even inside the "link", I get "Must specify index in a list subquery, not a unique subquery" Thoughts? I need to structure my query inside a list because I need the cursor.... Alec From alecf at metaweb.com Mon Apr 2 22:44:37 2007 From: alecf at metaweb.com (Alec Flett) Date: Mon, 02 Apr 2007 15:44:37 -0700 Subject: [Developers] "index" on any_reverse/any_master In-Reply-To: <53815B81-A71D-438C-9342-37CBAFBC1215@metaweb.com> References: <46117947.8070100@metaweb.com> <53815B81-A71D-438C-9342-37CBAFBC1215@metaweb.com> Message-ID: <46118755.9040006@metaweb.com> Unfortunately, I need the actual index that links ...#9202a8c04000641f800000000029cf71 to the outer links along the written_by property... This is what it looks like inverted, doing what I want: {"id": "#9202a8c04000641f800000000029cf71", "/film/film/written_by": [{ "timestamp": null, "index":null, "sort": "index" }] } But I can't invert it because I need a cursor. Alec > > does this do what you want? > > { > "q":{ > "cursor":true, > "query":[{ > "/type/reflect/any_reverse":[{ > "id":"#9202a8c04000641f800000000029cf71", > "index":null, > "link":{ > "master_property":"/film/film/written_by", > "timestamp":null > } > }], > "id":null, > "limit":30, > "sort":"/type/reflect/any_reverse.link.timestamp" > }] > } > } > From tim at metaweb.com Tue Apr 3 00:13:32 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 2 Apr 2007 17:13:32 -0700 Subject: [Developers] Weekly Sandbox refresh Message-ID: Every Monday, usually between 4pm and 6pm Pacific Time (2300-0100 UTC), we erase all data from sandbox.freebase.com and update it with a fresh snapshot from www.freebase.com. This week, we're running a little late; we expect the update to occur sometime in the next two hours. Tim Kientzle Metaweb Technologies, Inc. From niallo at metaweb.com Tue Apr 3 00:38:24 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 02 Apr 2007 17:38:24 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: References: Message-ID: <4611A200.3020504@metaweb.com> The sandbox data refresh has just been completed. Tim Kientzle wrote: > Every Monday, usually between 4pm and 6pm Pacific Time (2300-0100 UTC), > we erase all data from sandbox.freebase.com and update it with a fresh > snapshot from www.freebase.com. > > This week, we're running a little late; we expect the update > to occur sometime in the next two hours. > > Tim Kientzle > Metaweb Technologies, Inc. > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers > -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From william at scissor.com Tue Apr 3 00:57:46 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 17:57:46 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611A200.3020504@metaweb.com> References: <4611A200.3020504@metaweb.com> Message-ID: <4611A68A.8010103@scissor.com> Is the sandbox site otherwise broken? I just get a blank screen when I log in. Thanks, William Niall O'Higgins wrote: > The sandbox data refresh has just been completed. > > Tim Kientzle wrote: > >> Every Monday, usually between 4pm and 6pm Pacific Time (2300-0100 UTC), >> we erase all data from sandbox.freebase.com and update it with a fresh >> snapshot from www.freebase.com. >> >> This week, we're running a little late; we expect the update >> to occur sometime in the next two hours. >> >> Tim Kientzle >> Metaweb Technologies, Inc. >> >> _______________________________________________ >> Developers mailing list >> Developers at freebase.com >> http://lists.freebase.com/mailman/listinfo/developers >> >> > > > From jg at metaweb.com Tue Apr 3 00:58:47 2007 From: jg at metaweb.com (John Giannandrea) Date: Mon, 2 Apr 2007 17:58:47 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611A68A.8010103@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> Message-ID: <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> works for me. -jg On Apr 2, 2007, at 5:57 PM, William Pietri wrote: > Is the sandbox site otherwise broken? I just get a blank screen when I > log in. From william at scissor.com Tue Apr 3 01:13:10 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 18:13:10 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> Message-ID: <4611AA26.1050801@scissor.com> Interesting. All the dynamic stuff is still broken for me. The only pages that work are static ones: http://sandbox.freebase.com/view/tutorial/ http://sandbox.freebase.com/view/showcase And on the home page, I get the sidebar, but nothing else. This happens for me in Firefox 1.5, even after restarting it. It also is what I get in IE 6, if that helps. William John Giannandrea wrote: > > works for me. > -jg > > On Apr 2, 2007, at 5:57 PM, William Pietri wrote: >> Is the sandbox site otherwise broken? I just get a blank screen when I >> log in. > From colin at metaweb.com Tue Apr 3 01:17:01 2007 From: colin at metaweb.com (Colin Evans) Date: Mon, 02 Apr 2007 18:17:01 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AA26.1050801@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> Message-ID: <4611AB0D.5000909@metaweb.com> Login is fubar as well - 'Error contacting server.' William Pietri wrote: > Interesting. All the dynamic stuff is still broken for me. The only > pages that work are static ones: > > http://sandbox.freebase.com/view/tutorial/ > http://sandbox.freebase.com/view/showcase > > And on the home page, I get the sidebar, but nothing else. > > This happens for me in Firefox 1.5, even after restarting it. It also is > what I get in IE 6, if that helps. > > William > > > John Giannandrea wrote: > >> works for me. >> -jg >> >> On Apr 2, 2007, at 5:57 PM, William Pietri wrote: >> >>> Is the sandbox site otherwise broken? I just get a blank screen when I >>> log in. >>> > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070402/1f2b5f10/attachment-0002.htm From faye at metaweb.com Tue Apr 3 01:19:25 2007 From: faye at metaweb.com (Faye Li) Date: Mon, 02 Apr 2007 18:19:25 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AA26.1050801@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> Message-ID: <4611AB9D.2020902@metaweb.com> Hi William, I think you're no longer logged in, and queries that need authentication (i.e. anything but the static content) are all failing. Try login again at: http://sandbox.freebase.com/signin/signin -- Faye William Pietri wrote: > Interesting. All the dynamic stuff is still broken for me. The only > pages that work are static ones: > > http://sandbox.freebase.com/view/tutorial/ > http://sandbox.freebase.com/view/showcase > > And on the home page, I get the sidebar, but nothing else. > > This happens for me in Firefox 1.5, even after restarting it. It also is > what I get in IE 6, if that helps. > > William > > > John Giannandrea wrote: > >> works for me. >> -jg >> >> On Apr 2, 2007, at 5:57 PM, William Pietri wrote: >> >>> Is the sandbox site otherwise broken? I just get a blank screen when I >>> log in. >>> > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070402/2ec9cbb8/attachment-0002.htm From william at scissor.com Tue Apr 3 01:23:10 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 18:23:10 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AB9D.2020902@metaweb.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> <4611AB9D.2020902@metaweb.com> Message-ID: <4611AC7E.5050500@scissor.com> Faye Li wrote: > Hi William, > > I think you're no longer logged in, and queries that need > authentication (i.e. anything but the static content) are all failing. > Try login again at: > > http://sandbox.freebase.com/signin/signin Logging in gets me the same result, that or bounced back to the signin screen. Clearing cookies and clearing cache have not helped appreciably either. I just discovered, though, that this happens only for one of my accounts. The account "querybuilder_bot" doesn't work, but "william" does. William From tim at metaweb.com Tue Apr 3 01:45:22 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 2 Apr 2007 18:45:22 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <4611AC7E.5050500@scissor.com> References: <4611A200.3020504@metaweb.com> <4611A68A.8010103@scissor.com> <98218177-A0C1-4AED-8B30-C02C0F7D7253@metaweb.com> <4611AA26.1050801@scissor.com> <4611AB9D.2020902@metaweb.com> <4611AC7E.5050500@scissor.com> Message-ID: This problem has been identified and fixed. When we copied the graph data over, the separate authentication database (which stores passwords and other private account data) did not get copied over. All account data from www.freebase.com has been copied over to sandbox.freebase.com, so accounts created in the last week on www.freebase.com now work correctly on sandbox.freebase.com. Tim Kientzle On Apr 2, 2007, at 6:23 PM, William Pietri wrote: > Faye Li wrote: >> Hi William, >> >> I think you're no longer logged in, and queries that need >> authentication (i.e. anything but the static content) are all >> failing. >> Try login again at: >> >> http://sandbox.freebase.com/signin/signin > > Logging in gets me the same result, that or bounced back to the signin > screen. Clearing cookies and clearing cache have not helped > appreciably > either. > > I just discovered, though, that this happens only for one of my > accounts. The account "querybuilder_bot" doesn't work, but > "william" does. > > William > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From william at scissor.com Tue Apr 3 02:49:42 2007 From: william at scissor.com (William Pietri) Date: Mon, 02 Apr 2007 19:49:42 -0700 Subject: [Developers] Query Builder 0.5 Message-ID: <4611C0C6.5060103@scissor.com> Thanks to all for feedback on Query Builder 0.4. Based on that, I bring you Query Builder 0.5: http://dev.scissor.com/querybuilder/ New features: * nested browsing and selection of type properties * filtering support for exact values of o dates o booleans o floating-point numbers o integers * the ability to open a query in the query editor That first one is particularly exciting to people. For example, you can now easily build a search for albums by groups in the same genre as The Police. If I counted right, that's seven clicks and a little typing. Known problems: * just as ugly as before * the type browser sometimes jumps back to the top * the "open on sandbox" button still works even for those not logged in to sandbox As before, suggestions and comments are warmly welcomed! William From vtalwar at berkeley.edu Tue Apr 3 04:33:16 2007 From: vtalwar at berkeley.edu (vtalwar at berkeley.edu) Date: Mon, 2 Apr 2007 21:33:16 -0700 (PDT) Subject: [Developers] mjt xhtml In-Reply-To: References: Message-ID: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> Hi all, The thought of using mjt to create svg's recently popped into my head. I tried making a simple example but a crippling problem arose (in Firefox 2.0) which seems to have to do with XML vs. HTML. I spent about a couple of hours looking through mjt.js for the source to no avail. Essentially what's happening is that when the attributes are generated for "top", you get something like this trying to be created in the XML document: (this markup is from the flatten_markup function) This is fine in the HTML document but is problematic for the XML document since it doesn't allow for duplicate attributes (the id attr). The end result is that setting the innerHTML value for tmpdiv in replace_html() has no effect (since the xml doesn't validate I guess? not sure...). I got around this problem by changing using createElementNS with "http://www.w3.org/TR/REC-html40" to create tempdiv and also the original "top" div but then document.getElementById doesn't seem to be able to find "top" (maybe since it has a different namespace now? not sure again...). Any suggestions would be much appreciated :) Thanks, Vishal From nix at metaweb.com Tue Apr 3 19:48:00 2007 From: nix at metaweb.com (Nick Thompson) Date: Tue, 03 Apr 2007 12:48:00 -0700 Subject: [Developers] mjt xhtml In-Reply-To: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> Message-ID: <4612AF70.9090603@metaweb.com> vtalwar at berkeley.edu wrote: > The thought of using mjt to create svg's recently popped into my head. neat! i'd thought about trying this but figured that xml vs html issues would make it too hard - but it sounds like you already got pretty far. if it's making it as far as the double id="..." attribute then i think the chances are good that it can work, at least in firefox. > Essentially what's happening is that when the attributes are generated for > "top", you get something like this trying to be created in the XML > document: > > there is a mjt.js bug here... as you point out, the browser tolerates the duplicated id="top" in html, but xml is not so forgiving. i'll be pushing out a new version of mjt.js this week with a fix for this bug as well as some other goodies. i need to test the new goodies on ie6 first :-/. nick > > (this markup is from the flatten_markup function) > This is fine in the HTML document but is problematic for the XML document > since it doesn't allow for duplicate attributes (the id attr). The end > result is that setting the innerHTML value for tmpdiv in replace_html() > has no effect (since the xml doesn't validate I guess? not sure...). I got > around this problem by changing using createElementNS with > "http://www.w3.org/TR/REC-html40" to create tempdiv and also the original > "top" div but then document.getElementById doesn't seem to be able to find > "top" (maybe since it has a different namespace now? not sure again...). > > Any suggestions would be much appreciated :) > > Thanks, > Vishal > > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From brendan at metaweb.com Tue Apr 3 20:22:22 2007 From: brendan at metaweb.com (Brendan Neutra) Date: Tue, 03 Apr 2007 13:22:22 -0700 Subject: [Developers] images and cc In-Reply-To: <4612AF70.9090603@metaweb.com> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> <4612AF70.9090603@metaweb.com> Message-ID: <4612B77E.2030102@metaweb.com> Hi all, A question on the data side of things: I'm developing some great architecture data and I've got a script to produce a google earth file from Freebase data (shameless plug: http://www.neutraweb.com/Richard_Neutra.kml) I've contacted some flickr users who have creative commons license photos that are of interest and have gotten really positive response from them. What's the proper way to attribute these kinds of photos? I know there is a "source" property for images/content. There is also the article you can attach to the photo. This also brings up the thorny issue of: if someone builds a 3rd party app, how do we encourage/push them to expose the attribution of the photo in the app? Brendan From jg at metaweb.com Wed Apr 4 00:27:57 2007 From: jg at metaweb.com (John Giannandrea) Date: Tue, 3 Apr 2007 17:27:57 -0700 Subject: [Developers] images and cc In-Reply-To: <4612B77E.2030102@metaweb.com> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> <4612AF70.9090603@metaweb.com> <4612B77E.2030102@metaweb.com> Message-ID: <20070404002757.GB32611@metaweb.com> Brendan Neutra wrote: > What's the proper way to attribute these kinds of photos? For the mass image import going on on sandbox, soon on the main site, I have been using /common/licensed_object properties on the image object. This includes an explicit statement of license. Provenance is much harder to model since oftentimes you dont know the origin. We need to revise the image template to display these items and allow them to be added after a manual image upload. Ill file a bug for this and for finalizing the schema for image and how it relates to topic and document. -jg From brendan at metaweb.com Wed Apr 4 00:44:32 2007 From: brendan at metaweb.com (Brendan Neutra) Date: Tue, 03 Apr 2007 17:44:32 -0700 Subject: [Developers] images and cc In-Reply-To: <20070404002757.GB32611@metaweb.com> References: <21636.75.24.208.107.1175574796.squirrel@calmail.berkeley.edu> <4612AF70.9090603@metaweb.com> <4612B77E.2030102@metaweb.com> <20070404002757.GB32611@metaweb.com> Message-ID: <4612F4F0.2090809@metaweb.com> here's some relevant info: http://creativecommons.org/learnmore/faq#How_do_I_properly_attribute_a_Creative_Commons_licensed_work? and my approach to complying: license: cc by-nc photographer: Ken Mccown source: http://www.flickr.com/photos/kenmccown/ notes: * for now I just put this in the image description, but I agree it should be stored in it's own place * It is probably safe to assume that the photographer would want to keep the same name of the piece (though if it's very uninformative one could put that in the alias) though you don't technically have to do this unless they specify it. * Rather then give the exact url source of the image (which I don't think has much chance of being maintained) I chose to specify the users flick home page as a source for what it's worth. I suppose this is a work in progress... John Giannandrea wrote: > Brendan Neutra wrote: > >> What's the proper way to attribute these kinds of photos? >> > > For the mass image import going on on sandbox, soon on the > main site, I have been using /common/licensed_object > properties on the image object. This includes an > explicit statement of license. Provenance is much harder > to model since oftentimes you dont know the origin. > > We need to revise the image template to display these items > and allow them to be added after a manual image upload. > Ill file a bug for this and for finalizing the schema > for image and how it relates to topic and document. > > -jg > From william at scissor.com Wed Apr 4 01:00:40 2007 From: william at scissor.com (William Pietri) Date: Tue, 03 Apr 2007 18:00:40 -0700 Subject: [Developers] Query Builder 0.5 In-Reply-To: <4611C0C6.5060103@scissor.com> References: <4611C0C6.5060103@scissor.com> Message-ID: <4612F8B8.9060301@scissor.com> William Pietri wrote: > Thanks to all for feedback on Query Builder 0.4. Based on that, I bring > you Query Builder 0.5: > > http://dev.scissor.com/querybuilder/ > I've released a new version of this; let's call it 0.51. The main difference is that property browser (the upper middle pane) now does not jump around like a barefoot child on hot pavement. For those wondering, the problem was apparently a a GWT bug: http://code.google.com/p/google-web-toolkit/issues/detail?id=369 William From william at scissor.com Thu Apr 5 01:17:58 2007 From: william at scissor.com (William Pietri) Date: Wed, 04 Apr 2007 18:17:58 -0700 Subject: [Developers] Querybuilder 0.6 Message-ID: <46144E46.7040103@scissor.com> Thanks again for the valuable feedback. I have released yet another version: http://dev.scissor.com/querybuilder/ This version adds range filtering for integers, floating point numbers and dates. I've rushed a little more than I like with this version, as I'll be on vacation for a few weeks starting Friday morning. If you notice issues, please let me know and I'll fix them promptly. Thanks, William From tim at metaweb.com Mon Apr 9 19:35:45 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 9 Apr 2007 12:35:45 -0700 Subject: [Developers] www.freebase.com operating read-only. Message-ID: We're experiencing a problem with www.freebase.com right now. The service has been degraded to read-only access until we can resolve it. Thank you for your patience, Tim Kientzle Metaweb Technologies, Inc. From mt at alum.mit.edu Mon Apr 9 20:02:08 2007 From: mt at alum.mit.edu (Mike Travers) Date: Mon, 9 Apr 2007 13:02:08 -0700 Subject: [Developers] Query worked last night, today it times out Message-ID: I made a simple page with mjt to show the most recently created objects, using this query: [{ "creator":{name: null}, "limit":100, "name":null, "id": null, "sort":"-timestamp", "timestamp":null, "type": [{name: null, id: null}] }] This worked fine until this morning, now it doesn't. It appears to be timing out (when I run it from the query editor page). Removing the "type" clause seems to fix the timeout, but I'd like to be getting that information. Maybe this is a bad query to be doing since it ranges over the whole universe. But it's strange that the behavior changed abruptly. It works OK on sandbox. Mike From tim at metaweb.com Mon Apr 9 20:40:30 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 9 Apr 2007 13:40:30 -0700 Subject: [Developers] www.freebase.com restored to normal operation. In-Reply-To: References: Message-ID: <24964209-67C4-4949-A6D2-7B3E43897911@metaweb.com> We've reset the master database and everything seems to be working again, so we've restored the system to read-write mode. As far as we can tell, no data was lost. If you notice any unusual behavior, please let us know. Thank you again for your patience, Tim Kientzle Metaweb Technologies, Inc. On Apr 9, 2007, at 12:35 PM, Tim Kientzle wrote: > We're experiencing a problem with www.freebase.com right now. > > The service has been degraded to read-only access until we can > resolve it. > > Thank you for your patience, > > Tim Kientzle > Metaweb Technologies, Inc. > > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From tim at metaweb.com Mon Apr 9 20:44:01 2007 From: tim at metaweb.com (Tim Kientzle) Date: Mon, 9 Apr 2007 13:44:01 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: References: Message-ID: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> We were having some system problems starting around 11am PDT. Those seem to be resolved now. Please try again and let us know if you're still having problems. Tim Kientzle On Apr 9, 2007, at 1:02 PM, Mike Travers wrote: > I made a simple page with mjt to show the most recently created > objects, using this query: > [{ > "creator":{name: null}, > "limit":100, > "name":null, > "id": null, > "sort":"-timestamp", > "timestamp":null, > "type": [{name: null, id: null}] > }] > > This worked fine until this morning, now it doesn't. It appears to be > timing out (when I run it from the query editor page). Removing the > "type" clause seems to fix the timeout, but I'd like to be getting > that information. > > Maybe this is a bad query to be doing since it ranges over the whole > universe. But it's strange that the behavior changed abruptly. It > works OK on sandbox. > > Mike > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From mt at alum.mit.edu Mon Apr 9 20:59:29 2007 From: mt at alum.mit.edu (Mike Travers) Date: Mon, 9 Apr 2007 13:59:29 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> References: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> Message-ID: I'm still getting the timeouts as before. Mike On 4/9/07, Tim Kientzle wrote: > We were having some system problems starting around 11am PDT. > Those seem to be resolved now. > > Please try again and let us know if you're still having > problems. > > Tim Kientzle > > On Apr 9, 2007, at 1:02 PM, Mike Travers wrote: > > > I made a simple page with mjt to show the most recently created > > objects, using this query: > > [{ > > "creator":{name: null}, > > "limit":100, > > "name":null, > > "id": null, > > "sort":"-timestamp", > > "timestamp":null, > > "type": [{name: null, id: null}] > > }] > > > > This worked fine until this morning, now it doesn't. It appears to be > > timing out (when I run it from the query editor page). Removing the > > "type" clause seems to fix the timeout, but I'd like to be getting > > that information. > > > > Maybe this is a bad query to be doing since it ranges over the whole > > universe. But it's strange that the behavior changed abruptly. It > > works OK on sandbox. > > > > Mike > > _______________________________________________ > > Developers mailing list > > Developers at freebase.com > > http://lists.freebase.com/mailman/listinfo/developers > > From mt at alum.mit.edu Mon Apr 9 21:30:43 2007 From: mt at alum.mit.edu (Mike Travers) Date: Mon, 9 Apr 2007 14:30:43 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: References: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> Message-ID: Well, now it's working, but there was a short period after your reboot where it wasn't. Mysterious. Mike On 4/9/07, Mike Travers wrote: > I'm still getting the timeouts as before. > > Mike > > On 4/9/07, Tim Kientzle wrote: > > We were having some system problems starting around 11am PDT. > > Those seem to be resolved now. > > > > Please try again and let us know if you're still having > > problems. > > > > Tim Kientzle > > > > On Apr 9, 2007, at 1:02 PM, Mike Travers wrote: > > > > > I made a simple page with mjt to show the most recently created > > > objects, using this query: > > > [{ > > > "creator":{name: null}, > > > "limit":100, > > > "name":null, > > > "id": null, > > > "sort":"-timestamp", > > > "timestamp":null, > > > "type": [{name: null, id: null}] > > > }] > > > > > > This worked fine until this morning, now it doesn't. It appears to be > > > timing out (when I run it from the query editor page). Removing the > > > "type" clause seems to fix the timeout, but I'd like to be getting > > > that information. > > > > > > Maybe this is a bad query to be doing since it ranges over the whole > > > universe. But it's strange that the behavior changed abruptly. It > > > works OK on sandbox. > > > > > > Mike > > > _______________________________________________ > > > Developers mailing list > > > Developers at freebase.com > > > http://lists.freebase.com/mailman/listinfo/developers > > > > > From william at scissor.com Mon Apr 9 22:52:17 2007 From: william at scissor.com (William Pietri) Date: Mon, 09 Apr 2007 15:52:17 -0700 Subject: [Developers] Query worked last night, today it times out In-Reply-To: References: <15C324FE-4C0A-4830-9008-4EC86C08575B@metaweb.com> Message-ID: <461AC3A1.6060306@scissor.com> Mike Travers wrote: > Well, now it's working, but there was a short period after your reboot > where it wasn't. Mysterious. > For what it's worth, I've seen similarly weird behavior with timeouts. I reported a couple, and then put the rest down to alpha stuff that will get fixed eventually. William From niallo at metaweb.com Mon Apr 9 23:49:23 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 09 Apr 2007 16:49:23 -0700 Subject: [Developers] Weekly Sandbox refresh Message-ID: <461AD103.2010805@metaweb.com> Our weekly refresh of the sandbox data will be happening over the next few minutes. -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From niallo at metaweb.com Mon Apr 9 23:58:27 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 09 Apr 2007 16:58:27 -0700 Subject: [Developers] Weekly Sandbox refresh In-Reply-To: <461AD103.2010805@metaweb.com> References: <461AD103.2010805@metaweb.com> Message-ID: <461AD323.50403@metaweb.com> Data refresh has been completed. Niall O'Higgins wrote: > Our weekly refresh of the sandbox data will be happening over the next > few minutes. > > -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From mjc at cs.washington.edu Wed Apr 11 20:45:12 2007 From: mjc at cs.washington.edu (Michael J Cafarella) Date: Wed, 11 Apr 2007 13:45:12 -0700 (PDT) Subject: [Developers] MQL and Metaweb data model Message-ID: Hi all, I've been reading through the MQL manual and am trying to figure out how MQL relates to other database query languages. I think I've figured a few things out, but I'd like to make sure I've got it right. A think Metaweb employees will be best-placed to answer a lot of this stuff, but it would be great to hear from anyone who has an opinion. Thanks, --Mike Cafarella ----------------------------------------------------------------- 1) It looks like there is no such thing as a "join" in the sense of creating a brand-new tuple as in a relational system. The only thing that can be returned by a query is a set of objects that already exist in the Metaweb store (or subparts of those objects). 2) However, it is very possible to perform semijoin selections. It's not hard to write a query that returns "all musical artists who are also movie stars". In fact, there seems to be no limit on the width of this semijoin. So it's possible to write a query that expresses: "Show tracks from all albums where the artist appears in a movie that was directed by someone born in city in a state with fewer than 1,000,000 residents" This query expresses a semijoin using the following sets: a) The set of tracks b) The set of albums c) The set of artists d) The set of movies e) The set of people (directors) f) The set of cities g) The set of states I imagine a query like this could be pretty burdensome to compute. 3) Are there implementation-specific limits on resource consumption, either on the number of objects returned or the size of the join? I know it defaults to a limit of 100, but there are no language-level limits in the docs that I see. 4) I can express a semijoin between any two arbitrary sets. Often these sets are all the instances of a given type, but they don't have to be. I think I could express a semijoin between albums and all objects that have "Police" in the name. Unlike the set of instances of a given type, this kind of set has not been materialized before I pose my query. Something like this also seems hard to compute. Do you expect the system to be able to handle this type of query at large scale? 5) It seems like XQuery could work fairly well on the Metaweb data model. Is there a reason you decided against it? Thanks again, --Mike From sm at metaweb.com Thu Apr 12 17:20:34 2007 From: sm at metaweb.com (Scott Meyer) Date: Thu, 12 Apr 2007 10:20:34 -0700 Subject: [Developers] MQL and Metaweb data model Message-ID: <461E6A62.8020503@metaweb.com> > From: Michael J Cafarella > 1) It looks like there is no such thing as a "join" in the sense of > creating a brand-new tuple as in a relational system. The only thing that > can be returned by a query is a set of objects that already exist in the > Metaweb store (or subparts of those objects). Unlike a relational database which deals only with values, identity is fundamental to Freebase. We work with sets of identities, not sets of values. For simplicity, MQL presents results as a nested series of dictionaries which, at a given level, look like semijoin results, but denormalizing these results in the obvious way can yield the table you'd expect from a natural join. There's actually no such thing as a semijoin either, the results just happen to look that way. :-) > 2) However, it is very possible to perform semijoin selections. It's not > hard to write a query that returns "all musical artists who are also movie > stars". In fact, there seems to be no limit on the width of this > semijoin. So it's possible to write a query that expresses: > > "Show tracks from all albums where the artist appears in a movie that was > directed by someone born in city in a state with fewer than 1,000,000 > residents" > > This query expresses a semijoin using the following sets: > > a) The set of tracks > b) The set of albums > c) The set of artists > d) The set of movies > e) The set of people (directors) > f) The set of cities > g) The set of states > > I imagine a query like this could be pretty burdensome to compute. Indeed. If we were doing traditional relational semijoins this would become intractable very quickly. We work directly with sorted sets of identities, which, like 4-wheel drive, allows us to get stuck further from civilization. If you push far enough, things do become computationally intractable, but the current system does surprisingly well on a variety of useful queries. And, we have only just begun to optimize. > 3) Are there implementation-specific limits on resource consumption, > either on the number of objects returned or the size of the join? I know > it defaults to a limit of 100, but there are no language-level limits in > the docs that I see. We limit queries to 8 seconds. There's currently no limit on memory consumption, but we do give up on queries that are "too hard." Note that there is cursor support for returning large datasets a page at a time. > 4) I can express a semijoin between any two arbitrary sets. Often these > sets are all the instances of a given type, but they don't have to be. I > think I could express a semijoin between albums and all objects that have > "Police" in the name. Unlike the set of instances of a given type, this > kind of set has not been materialized before I pose my query. > > Something like this also seems hard to compute. Do you expect the > system to be able to handle this type of query at large scale? Yes. Aside from their identity, the underlying tuple store is completely ignorant of "type" so the "set of instances of a given type" is no different from the "set of all instances with 'Police' in the name" The fact that relational stores conflate 'type' with physical storage - a sore point with Codd, I believe - makes it exceedingly difficult to use a relational store to do what Freebase does. This is one of the reasons that we built our own tuple store. > 5) It seems like XQuery could work fairly well on the Metaweb data model. > Is there a reason you decided against it? If the adoption of XPath or any other particular query syntax were critical to doing what Freebase does, we would have adopted it. We're exploring a new way to store and retrieve data, and the cost of implementing a standard syntax (or the inevitable super/sub/side-set of a standard syntax) far outweighs the benefits. As things evolve, we're completely open to supporting other syntaxes, or helping third parties to do so. The objective of MQL was to present a simple, stable subset of the underlying graph query functionality in a way that was palatable to both programmers and programming languages such as Javscript and Python. Regards, -Scott From mjc at cs.washington.edu Thu Apr 12 18:41:37 2007 From: mjc at cs.washington.edu (Michael J Cafarella) Date: Thu, 12 Apr 2007 11:41:37 -0700 (PDT) Subject: [Developers] MQL and Metaweb data model In-Reply-To: <461E6A62.8020503@metaweb.com> References: <461E6A62.8020503@metaweb.com> Message-ID: Hi Scott, Thanks for your detailed reply. I really appreciate the time you took. It sounds like your database is actually more similar to a search engine's document-store than to a relational database. In a search engine, documents are "identities" which have a bag of associated values. You could imagine marking a document with text that says "band=The_Police" or "type=Album". You could retrieve results for either the "band" query or the "type" query easily. The "wide-semijoin" that I described would entail looking up a number of these different sets, sorting them, then performing an intersection-style operation between them. Like you say, this would be difficult eventually (say, 1000 different types) but you might not run into problems for any real-world queries. This is very interesting stuff. I hope you consider publishing an article about it someday. --Mike From alecf at metaweb.com Fri Apr 13 19:04:31 2007 From: alecf at metaweb.com (Alec Flett) Date: Fri, 13 Apr 2007 12:04:31 -0700 Subject: [Developers] Flash Data Mobs Message-ID: <461FD43F.10404@metaweb.com> Internally at Metaweb, we've been running an almost-weekly "flash data mob" activity. The idea has been to take 5-10 minutes (or more, if you've got the inclination) and fill in some missing data in Freebase. The idea is for a bunch of people to all focus on flushing out one corner of data - the combined power of a bunch of people all doing a little work is really rather impressive. For an example of one of our first Flash Data Mobs, check out Kevin Bacon: http://www.freebase.com/view?id=/wikipedia/en/Kevin_Bacon His entry is very well filled out - not only as an actor but as a person, a director, even a reader of audio books! (Feel free to add more, but keep reading!) A lot of great things come out of this: 1) Unusual issues with the user interfaces - sometimes bugs, and sometimes real problems with the way we present and edit the data. 2) Data modelling issues - there was lots of discussion about what it meant for Kevin to play "himself" on Will and Grace some time back 3) Data source questions - IMDB is great for getting movie/tv data, but what about music? books? biographical information? You get the idea and of course 4) even more quality data in Freebase. This is great for developers because we learn about the data and get inspiration for new mashups. So I'm going to be running a weekly Flash Data Mob - each week we'll have some interesting task or topic that we'll all try to fill out, and we'll take up the discussion on this list. Later today I'll send out our first public Flash Data Mob activity that should give you something fun to do for a lazy friday afternoon.. stay tuned! I'll also be posting this and the first flash mob on our blog at http://blog.freebase.com/ Alec From alecf at metaweb.com Sat Apr 14 02:37:37 2007 From: alecf at metaweb.com (Alec Flett) Date: Fri, 13 Apr 2007 19:37:37 -0700 Subject: [Developers] Friday the 13th Data Flash Mob Message-ID: <46203E71.7050403@metaweb.com> Did I say "Lazy Friday" - I should have said "Friday the 13th" - last minute stuff delayed this a bit, sorry about that. Anyway, here's our first public Data Flash Mob. If you missed my earlier message, see the blog post: http://blog.freebase.com/?p=20 And in the spirit of Friday the 13th, we're going to fill out the Friday the 13th Films! Start here: http://www.freebase.com/view?id=%239202a8c04000641f8000000000134e8e If we all just take a few minutes to fill out the original film and the outgoing links, we can probably connect all of its sequels together pretty easily - not just by film series but also by actors/etc that appeared in the various films. I read recently that New Line Cinema recently acquired rights to this series, so it would be interesting to see if we can do a kevin-bacon-game style link between the original movie and New Line.. see what their connection is/was before they produce the next film. So the goal here is to fill out the Friday the 13th series of films to get the shortest link between New Line - i.e. maybe the directors of one of the sequels directed other movies by New Line? Who knows! New Line seems to be incorrectly typed as Film, so we should fix that too http://www.freebase.com/view?id=%239202a8c04000641f8000000000225dd9 Good luck! Please report back to the list with any feedback, questions, comments, etc... Alec From jonathanicity at gmail.com Sun Apr 15 00:06:15 2007 From: jonathanicity at gmail.com (Siddharth Jonathan) Date: Sat, 14 Apr 2007 17:06:15 -0700 Subject: [Developers] freebase data download Message-ID: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> Hi, I'm new to freebase and I'm still looking around. Can someone tell me what the quickest way is to download all the freebase data? I would like to do that to analyze some of the data. If you can point me to any scripts/information that would be very helpful. Thanks! Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070414/9314fb72/attachment-0002.htm From jg at metaweb.com Sun Apr 15 02:30:59 2007 From: jg at metaweb.com (John Giannandrea) Date: Sat, 14 Apr 2007 19:30:59 -0700 Subject: [Developers] freebase data download In-Reply-To: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> References: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> Message-ID: <370F950D-C553-424B-AB9F-8F6E2AEABB18@metaweb.com> Hi The easiest way to do it is by type, asking for all the instances of a specific type and whatever properties you are interested in. You need to use the cursor feature of MQL to make very large downloads. What language do you want to do this in, we may have something that we can provide that already does this. If you want to download absolutely everything in some specific format, contact me off list and we can discuss the options. -jg On Apr 14, 2007, at 5:06 PM, Siddharth Jonathan wrote: > Hi, > I'm new to freebase and I'm still looking around. > Can someone tell me what the quickest way is to download all the > freebase data? > I would like to do that to analyze some of the data. > If you can point me to any scripts/information that > would be very helpful. > Thanks! > > Jonathan > _______________________________________________ > Developers mailing list > Developers at freebase.com > http://lists.freebase.com/mailman/listinfo/developers From jonathanicity at gmail.com Sun Apr 15 02:33:01 2007 From: jonathanicity at gmail.com (Siddharth Jonathan) Date: Sat, 14 Apr 2007 19:33:01 -0700 Subject: [Developers] freebase data download In-Reply-To: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> References: <616305cc0704141706i1ac1c284h4466af830f12eccd@mail.gmail.com> Message-ID: <616305cc0704141933k7d0d4f22rf6d74742a706c008@mail.gmail.com> I guess one simple way appears to be to get all the guids out in pass, and in a second pass, pull out every property for each object referenced by each guid using a wildcard. Jonathan On 4/14/07, Siddharth Jonathan wrote: > > Hi, > I'm new to freebase and I'm still looking around. > Can someone tell me what the quickest way is to download all the freebase > data? > I would like to do that to analyze some of the data. > If you can point me to any scripts/information that would be > very helpful. > Thanks! > > Jonathan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.freebase.com/pipermail/developers/attachments/20070414/2e956590/attachment-0002.htm From niallo at metaweb.com Mon Apr 16 23:54:24 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 16 Apr 2007 16:54:24 -0700 Subject: [Developers] Weekly sandbox.freebase.com data refresh Message-ID: <46240CB0.1020104@metaweb.com> Hi, The weekly data refresh & software update of sandbox.freebase.com will be starting shortly. -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From niallo at metaweb.com Tue Apr 17 00:52:05 2007 From: niallo at metaweb.com (Niall O'Higgins) Date: Mon, 16 Apr 2007 17:52:05 -0700 Subject: [Developers] Weekly sandbox.freebase.com data refresh In-Reply-To: <46240CB0.1020104@metaweb.com> References: <46240CB0.1020104@metaweb.com> Message-ID: <46241A35.9040508@metaweb.com> Completed. Niall O'Higgins wrote: > Hi, > > The weekly data refresh & software update of sandbox.freebase.com will > be starting shortly. > > -- Niall O'Higgins Software Engineer Metaweb Technologies, Inc. From jack at metaweb.com Thu Apr 19 23:59:12 2007 From: jack at metaweb.com (Jack Alves) Date: Thu, 19 Apr 2007 16:59:12 -0700 Subject: [Developers] date constraint by month-day only Message-ID: What is the best way to get topics with a date that includes a specific month and day? I was given the query below to get all topics with a specific full date. MQL docs specify numeric constraint operators for dates. So the only way I can think of to get anything that happened on any Nov 22 is to repeat the query for every year. Is there a more efficient way? Will MQL eventually support a date contraint like "*-11-22"? Here's the query I'm using, { "query":[{ "/type/reflect/any_value":[{ "link":{ "master_property":{ "name":null } }, "type":"/type/datetime", "value":"1963-11-22" }], "id":null, "name":null, "sort":"/type/reflect/any_value.link.master_property.name", "type":"/common/topic" }] } Here's a link to a mjt script that uses the above query, http://dev.mqlx.com/%7Ejack/mjt/dayHistory.html?year=1963&day=11-22 Here's a link to the queryeditor with the above query, http://www.freebase.com/view/queryeditor/?q={%20%22query%22:[{%20%22/type/re flect/any_value%22:[{%20%22link%22:{%20%22master_property%22:{%20%22name%22: null%20}%20},%20%22type%22:%22/type/datetime%22,%20%22value%22:%221963-11-22 %22%20}],%20%22id%22:null,%20%22name%22:null,%20%22sort%22:%22/type/reflect/ any_value.link.master_property.name%22,%20%22type%22:%22/common/topic%22%20} ]%20%7d From jg at metaweb.com Fri Apr 20 01:45:19 2007 From: jg at metaweb.com (John Giannandrea) Date: Thu, 19 Apr 2007 18:45:19 -0700 Subject: [Developers] date constraint by month-day only In-Reply-To: References: Message-ID: <93F6AFAF-4D14-4AA9-9900-70F6D09113EC@metaweb.com> You cant currently use ~= with datetime. It used to work, we should reconsider it. -jg Jack Alves wrote: > What is the best way to get topics with a date that includes a > specific > month and day? I was given the query below to get all topics with a > specific > full date. MQL docs specify numeric constraint operators for dates. > So the > only way I can think of to get anything that happened on any Nov 22 > is to > repeat the query for every year. Is there a more efficient way? > Will MQL > eventually support a date contraint like "*-11-22"? From alecf at metaweb.com Mon Apr 23 17:23:02 2007 From: alecf at metaweb.com (Alec Flett) Date: Mon, 23 Apr 2007 10:23:02 -0700 Subject: [Developers] Flash Data Mob #2: Spinoffs Message-ID: <462CEB76.7090309@metaweb.com> I found this really cool page on Wikipedia: http://en.wikipedia.org/wiki/List_of_television_spin-offs This is a whole list of all the shows that have spun off of other shows. Many people have been painstakingly editing this page to keep this accurate and in sync with their respective articles, but this is an area where Freebase really shines - once the data is in there, it's entirely possible to render this page programmatically. So the challenge here is a two parter - one for the TV geeks to enter as much of this data as possible, and one for some MJT hackers. Here are the details: 1) Look through this list and find at least 5 shows with spin-offs, and connect them up in freebase. Just search for their titles in the search box that's on every freebase page. I've already hooked up the Happy Days chain (who knew Happy Days itself was a spin-off?) http://www.freebase.com/view?id=%239202a8c04000641f80000000001379b3 To get you started, here are a few of the more interesting "roots" of these spinoff trees.. you'll find some real gems even if you don't recognize the root show... Looney Toons: http://www.freebase.com/view?id=%239202a8c04000641f8000000000062641 Sam and Friends: http://www.freebase.com/view?id=%239202a8c04000641f80000000003fccf6 Broadway Open House: http://www.freebase.com/view?id=%239202a8c04000641f800000000019716d Cheers: http://www.freebase.com/view?id=%239202a8c04000641f800000000000e101 Diff'ren Strokes: http://www.freebase.com/view?id=%239202a8c04000641f800000000005bbc2 Law & Order: http://www.freebase.com/view?id=%239202a8c04000641f800000000013b473 (and yeah, we're fixing that '&' problem!) All in the Family: http://www.freebase.com/view?id=%239202a8c04000641f800000000012d68a 2) The MJT challenge see if you can render the equivalent of the Wikipedia page with MJT. This is a good test of MQL and MJT - how easy is it to query AND render this multi-level hierarchy? This is potentially a weak part of MQL, but I'd love to see how some creative folks approach the problem. Alec From tsturge at metaweb.com Mon Apr 23 17:40:43 2007 From: tsturge at metaweb.com (Tim Sturge) Date: Mon, 23 Apr 2007 10:40:43 -0700 Subject: [Developers] date constraint by month-day only In-Reply-To: References: Message-ID: <462CEF9B.30009@metaweb.com> This is a hard problem with MQL as it currently stands. There's no good solution right now beyond asking for everything and filtering or asking for each year in a separate query; neither work very well. I quite like the syntax "*-11-22" but it would need a fair bit of work internally (firstly so as not to be horribly inefficient, and secondly so as not to match 11:22 AM as well -- the current ~= syntax implies that - and : are equivalent) I'm going t