2017-07-19 Carlos Garnacho <carlosg@gnome.org> Release 1.99.1 build: Cleanup TRACKER_COMPILATION defines from c_args It is now defined globally, so can removed from specific targets. build: Unbreak meson build The change in 1b31260292 added somewhat clumsy meson support. The C-side vapi is now required by the Vala-side vapi, but only the latter is included when building the rest of the libraries. Since we can't tell the build to ditch the vala vapi and include our merged/spiced up tracker-sparql-2.0.vapi file, add a dependency on the C vapi that we can add on the selected places. A side effect of including the unfixed vapis is that includes point to internal files, so make sure everything gets TRACKER_COMPILATION when building to circumvent it. libtracker-sparql: Fix tracker-sparql-2.0.vapi generation It would be left partly including headers that must not be accessed directly. Also, ensure that we look for the C vapi file in srcdir for both meson and autotools. libtracker-data: Implement tracker:title-order() SPARQL function This is a tracker-specific helper function that will remove the initial common articles (the, a, an, ...) for sorting purposes. This is implemented as an additional collation, so tracker:title-order only makes sense to be used in the ORDER BY clause. 2017-07-18 Fabio Tomat <f.t.public@gmail.com> Update Friulian translation 2017-07-17 Rico Tzschichholz <ricotz@ubuntu.com> libtracker-miner: Fix introspection warning due header/source-def mismatch 2017-07-16 Carlos Garnacho <carlosg@gnome.org> tracker-store: Check wal threadpool existence before using it During shutdown the thread pool might be actually gone. Do not try to poke it in those situations, a blocking wal checkpoint will happen eventually. 2017-07-16 Emmanuele Bassi <ebassi@gnome.org> build: Add generated enumeration files to BUILT_SOURCES Dependency tracking in Automake is a side effect of compilation. This means two things: * headers are ignored when it comes to building dependencies, even when listed inside sources for a binary target * generated headers included inside source files are not used as a dependency until compilation of the source file completes successfully If binary target L depends on a source file C, and that source file includes a generated header H, there is no guarantee that the header H will be available by the time source file C is built, because the dependency is extracted from C itself, not from the list of sources. Additionally, any header H does not count towards the dependency resolution because headers are only used for ancillary features, like extracting tags. For this reason, Automake has `BUILT_SOURCES`. This is a bit of obscure corner of Automake that trips up many who are not practitioners of the Dark Arts. More information is available here: https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example 2017-07-15 Carlos Garnacho <carlosg@gnome.org> tracker-miner-fs: Fix signal prototype of TrackerMinerFiles::writeback It's defined a boolean return value, which was not obtained when emitting the signal. Agree everywhere on removing it since it's not necessary. Fixes a crash when the signal was emitted. libtracker-data: Do not reuse TrackerDBInterface used for DB checks It is just too soon to let it initialize like the others, just drop it and let the TrackerDBManager create proper ones on demand. Fixes possible crashes on FTS because the TrackerOntology is still empty, so the TrackerDBInterface ends up with 0 cached FTS properties. https://bugzilla.gnome.org/show_bug.cgi?id=784926 2017-07-13 Dušan Kazik <prescott66@gmail.com> Update Slovak translation 2017-07-12 Carlos Garnacho <carlosg@gnome.org> libtracker-miner: Disable monitors during crawling on large indexing trees If the number of indexed folders exceed the TrackerMonitor limit, acknowledge that the filesystem image shall be possibly out of date soon and disable monitors temporarily during crawling. This results in a major speedup in the crawling phase (both initial indexing and the one performed on every startup to update the Tracker concept of the filesystem), because an excessive amount of directory monitors make the syscalls performed during crawling inherently slower. Testing with the same sample folders than commit 2e7e70a2ea (and the same quarter million files in 17k folders), this results on ~4x faster tracker-miner-fs startups. The accumulated improvements took regular startup stituations (i.e. just ensuring the filesystem is up-to-date) from roughly 4m to 22s here. Everything measured on i7/ssd, otherwise idle, ... libtracker-miner: Add TrackerMonitor function to check the limit 2017-07-11 Carlos Garnacho <carlosg@gnome.org> libtracker-data: Do not accept "... AS foo" anymore Nowadays this is pretty useless nonstandard syntax, SPARQL1.1 settled on ".... AS ?foo" and we've accepted it for a long time. libtracker-sparql: Implement tracker_sparql_connection_get_namespace_manager() This is implemented on direct connections and the generic backend. The TrackerNamespaceManager will be filled in from the internal ontologies, as read by the TrackerDataManager. libtracker-data: Add tracker_data_manager_get_namespaces() This will be used to fill in TrackerNamespaceManager objects that are specific to a TrackerSparqlConnection. libtracker-sparql: Add vapi file for the C portions Just like we do with girs, merge together the vapi for the C portions and the generated vapi for the vala bits. This allows using TrackerNamespaceManager and TrackerResource from vala code. libtracker-sparql: Deprecate TrackerSparqlBuilder It has been superseded by TrackerResource, it is encouraged for new development. libtracker-miner: Drop TrackerSparqlBuilder from TrackerDecorator API Just add tracker_decorator_info_complete[_error] functions to let the decorator implementation give back the resulting SPARQL. This way the TrackerDecorator API is agnostic of the method/API used to build the SPARQL. libtracker-sparql: Fix horrible confusion with G_GINT64_FORMAT The "d" is already implied there, not for us to add. Let's pretend this never happened... libtracker-miner: Remove deprecation annotation from vapi file This applied to ignore_next_update(), which has been removed for 2.0. libtracker-sparql: Add tracker_resource_[add|set]_take_relation() These are C helper functions, so users may create trees of TrackerResources without having to manually unref all non-root ones. docs: Remove doc references to TrackerEnumerator libtracker-sparql: Remove get_direct(_async|_finish) Those functions have been deprecated for years and are being removed for 2.0. Use tracker_sparql_connection_get() and async variants instead, that will smartly use the direct connection underneath. 2017-07-10 Jeremy Bicha <jbicha@ubuntu.com> Fix Spanish translation header See https://bugzilla.gnome.org/show_bug.cgi?id=771765 2017-07-10 Kukuh Syafaat <syafaatkukuh@gmail.com> Update Indonesian translation 2017-07-10 Carlos Garnacho <carlosg@gnome.org> tests: Adapt tracker-backup-test to non-GPrivate interfaces This code relies on being able to query again the contents immediately after restore, this may or may not be true depending on the TrackerDBInterface used on the posterior query (eg. not being the one that performed the update). Since testing access from multiple DBInterfaces is not in the scope of this test, just add an extra TrackerDataManager initialization in between to ensure the interface used on the query is up to date. libtracker-data: Don't use GPrivate to store TrackerDBInterfaces The improvement is twofold here. On one hand ensures there's no confusions with multiple TrackerDBManagers, because we had no guarantees that a thread would be reused across those. But also we can be much smarter about which TrackerDBInterface we hand for operations, giving one that is "free" (i.e. has no other active cursors) gives much more reduced mutex contention on stress loads. The maximum number of interfaces has been obtained rather unscientifically, seems the turning point at which throughtput does not really increment on downright stupid workloads, given the amount of readonly threads we allow on the direct connection. libtracker-data: Make all other ifaces than wal/writable readonly The interfaces returned by tracker_data_manager_get_db_interface() are now ensured to be only used in select queries, so be bolder about this and ensure the interfaces are actually readonly. libtracker-data: Make writable/wal interfaces unique Instead of "picking" one of the several DB interfaces created, ensure canonical ones for these operations. libtracker-data: Pass DB interface to tracker_data_query_resource_id() This is used on both readonly and readwrite paths, pass the DB interface explicitly to avoid confusions wrt the one that must be used. libtracker-miner: Fix cancellation of data_provider_begin() The async op is not owner of the user data, so it may be actually gone in the GAsyncReadyCallback. Ensure we only use it on success or on other errors than cancelled, fixes crashes in tracker-crawler crawl-interrupted test after commit b4f7b84fe4 because tracker_crawler_stop() is now called at a time that triggers this. 2017-07-10 Piotr Drąg <piotrdrag@gmail.com> Update POTFILES.in 2017-07-10 Carlos Garnacho <carlosg@gnome.org> libtracker-miner: Ensure sparql buffer keeps flushing The sparql buffer might get full again with new tasks before the update_array operation for the current batch returned. In this case nothing will kick the TrackerMinerFS again, nor the SPARQL buffer from flushing again. Fix this by just flushing again, the miner will follow as soon as the SPARQL buffer is below limits. libtracker-data: Set interfaces' prepared statement LRUs on common code All DB interfaces get the max number of items set, so just set those in common code. Also, avoid allocating any space for update statements in readonly interfaces. libtracker-miner: Call next_files_async() in batches Instead of painstakingly requesting GFileInfos one by one, request several of those in a single batch. This is a lot easier to the main loop, because we don't need hitting it several times per file. This and the previous commits result in massive speedups to TrackerMinerFS, since CPU time in userspace during TrackerMinerFS activity was largely dominated by GSource and main loop overhead. Somewhere around 1.80x-2x faster for initial indexing, and up to 3x faster for startup on an already indexed FS, from unscientifical testing with a quarter million files contained in ~17K folders. libtracker-miner: Remove Tracker[File]Enumerator This is a pretty pointless shallow wrapper to GFileEnumerator. Just make TrackerDataProvider offer GFileEnumerators. If anyone ever was in need for implementing a TrackerEnumerator, they can do the very same by implementing GFileEnumerator, and handing those from their TrackerDataProvider impl. This also cuts a level of indirection in directory crawling (i.e. no wrapping of GTasks in GTasks), and allows us to fully use GFileEnumerator API (eg. g_file_enumerator_next_files will be handy). 2017-07-09 Carlos Garnacho <carlosg@gnome.org> libtracker-miner: Process TrackerMinerFS elements in batches This prevents us from hitting too hard the main loop. libtracker-miner: Shortcut the initial op after tracker_crawler_start() It is well known, and will already happen asynchronously, no need to hit the main loop for that. libtracker-miner: Process TrackerCrawler elements in batches This prevents us from hitting too hard the main loop. libtracker-data: Preserve ontology journal writer for a bit longer It's destroyed far too soon if the ontology needs rebuilding.