.. _tour: A Tour of the DSL ================= The Dimensions Search Language (DSL) allows you to write concise expressions, called :term:`queries `, to search and retrieve information from the Dimensions dataset. This page provides an interactive walkthrough of DSL query basics, to get you familiar with the language. .. exercise:: In this tutorial, suggested interactive exercises to build up your DSL knowledge will appear in boxes like this. This is the first one! A simple query -------------- Let's start with a simple query example. One of the most basic and commonly-used types of questions we could ask about Dimensions data is something like, "Which are the most recent scholarly works related to a specific research topic?". We can answer such a question with a query like this, which asks: "Which are the most recent scientific publications related to malaria?" .. code-block:: none search publications for "malaria" return publications Whitespace is ignored in DSL queries, so feel free to use line breaks and indentation as you wish. The following query is exactly the same as the query above: .. code-block:: none search publications for "malaria" return publications .. _anatomy: The anatomy of a query ---------------------- Each DSL query is composed of two main parts: * A ``search`` phrase, e.g. ``search publications for "malaria"``, and * At least one ``return`` phrase, e.g. ``return publications`` The first part of the query, the ``search`` phrase, specifies the *set of documents* that we want to know something about. In the example ``search publications for "malaria"``, we are saying "I want to know something about publications related to malaria". ``search`` part can optionally specify whether to search using full text search or search only in title and abstract. This is accomplished by specifier ``in [full_data | title_abstract_only]`` that follows the source to search in, such as for example ``search grants in title_abstract_only``. The second part of the query, the ``return`` phrase, specifies *what we want to know* about the documents specified in the ``search`` phrase. In the example ``return publications``, we are saying "I want to see basic information about the publications matched by the ``search`` phrase". A DSL query is not complete without both a ``search`` phrase and at least one ``return`` phrase; we must tell the DSL both *which* data we are interested in and *what* we want to see from that data. Trying to run a query that has only one phrase or the other will result in an error, as you can see for yourself by trying to run these incomplete queries: .. code-block:: none search publications for "malaria" .. code-block:: none return publications In the following sections we'll explore what we can do with each of these two phrases. ``search`` for source documents ------------------------------- As we've just seen, the ``search`` phrase specifies a *set of documents* that will provide the data for the results we'll ask for in the ``return`` phrase. As all Dimensions data must be taken from one of the supported sources, this phrase always starts with the word ``search`` followed by the name of the desired source, e.g. ``search publications``. This is the only part of the ``search`` phrase that is not optional; the following is a valid (though perhaps not very useful) query where the ``search`` phrase specifies the set of all publications: .. code-block:: none search publications return publications Choose a document source ++++++++++++++++++++++++ In the previous queries we've searched for ``publications``, which is an example of a Dimensions data :term:`source`, i.e. a type of scientific document/work that may be of interest. Others include ``grants``, ``patents``, ``clinical_trials``, ``researchers`` and ``policy_documents``. We can query another source by replacing ``publications`` with the name of another source, like ``grants``. For example: .. code-block:: none search grants for "malaria" return grants .. exercise:: Try changing the query above to search for another type of work, e.g. ``patents``. (Hint: make sure you update both the ``search`` and ``return`` phrases; what happens when you only change one?) For a list of all currently supported sources, see the :ref:`data-sources` page. .. _searching-for: Search for research topics ++++++++++++++++++++++++++ As we've seen in the preceding examples, we're often interested in scientific work related to a particular topic, e.g. malaria. In the ``for`` phrase, we can specify such a topic as one or more :term:`search term` that we want the retrieved documents to match. We can specify a multi-word phrase as well, for example: .. code-block:: none search grants for "attention deficit disorder" return grants .. exercise:: Try modifiying the query above to search for different terms/subjects. Take a look at the titles of the returned documents to see how they may be related to the subject(s) you searched for. How does this type of search work behind the scenes? First, the complete text of all documents is analyzed, and each document is assigned a score that represents how well it matches the given terms(s). Documents are then ranked by their score, with the best-matching documents appearing first in the results. .. _searching-where: Restrict the search with filters ++++++++++++++++++++++++++++++++ We can restrict the set of documents matched by the ``search`` phrase by specifying certain filters that must apply to the retrieved documents, in other words certain properties these documents must have. We specify these as :ref:`filter expressions ` in the ``where`` phrase, as follows: .. code-block:: none search grants where start_year>=2010 and funders.acronym="DFG" for "attention deficit disorder" return grants The query above only matches documents that were published in the year 2010 or later and where the supporting funder had the acronym "DFG". .. exercise:: Try to modify the query above to match documents published *only* in 2010 (not in later years), then to match documents published in 2010 *or earlier*. (You can probably guess what operators you'd need to use instead of ``>=``, but consult the list of :ref:`filter operators ` if you get stuck.) Filters in the ``where`` phrase may also be used to retrieve one or more specific documents using an identifier such as a publication DOI, like so: .. code-block:: none search publications where doi in ["10.1186/s12888-017-1463-3", "10.1186/s40479-017-0057-5", "10.1186/s12888-017-1222-5"] return publications The :term:`fields `, :term:`entities `, etc. that may appear in filter expressions depend on the :term:`source` being searched; in the queries above, we see the ``year`` field as well as the ``funder`` entity's ``acronym`` field being used to search ``grants``, and the ``doi`` field being used to search ``publications``. If an unsupported field name is used, the DSL will report an error message to that effect, and provide a list of the supported fields to help you find the correct field name. For example, if you want to find grants that started in the year 2010 and create a filter for this using a field called ``startyear``, this will result in an error, because the field name is not quite correct. Try it for yourself - this query will raise an error: .. code-block:: none search grants where start_year=2010 return grants .. exercise:: Fix the query above to use the correct field name (hint: it's in the error message). Consult the documentation for the :ref:`where ` phrase (including the types of expressions/operators that may be used) to find out more about the types of filters and operators you can use, and the lists of supported fields for each :ref:`supported source ` to see which :term:`field` and :term:`entity` names can be used in filter expressions. .. exercise:: Play with the queries above to use other filter expression(s) that narrow the set of documents in different ways. .. note:: As you can see in the various queries above, the ``where`` phrase may be used in combination with a ``for`` phrase, or by itself. If both phrases are used, they can appear in either order - it does not matter which comes first. Behind the scenes, first the ``where`` filters will be applied to narrow down the set of documents, and then this restricted set of documents will be scored & ranked against the ``for`` search terms as described :ref:`above `. ``return`` information about documents -------------------------------------- As we said :ref:`earlier `, the ``search`` phrase allows us to specify *which documents* we're interested in, but it doesn't allow us to specify exactly *what we want to know* about those documents; for that we use the ``return`` phrase(s). A return phrase comes after the ``search`` phrase, and always starts with the word ``return`` followed by a specification of the result we wish to see. In the queries we've seen so far, this has been only the most basic type of result: the source documents themselves, i.e. ``return publications`` or ``return grants``. However, the ``return`` phrase gives us the flexibility to be much more specific about what we wish to see; in the following sections we'll explore some of the possibilities. Return source documents +++++++++++++++++++++++ If we are interested in knowing something about the :term:`source` documents themselves, we can ask the DSL to show us only certain information (metadata) about each document by indicating the specific :term:`field(s) ` we wish to see, like so: .. code-block:: none search publications for "malaria" return publications [doi + title + year] As in the ``search`` phrase, the :term:`field(s) ` that may appear in such a ``return`` phrase depend on the :term:`source` being searched/returned. .. exercise:: Modify the query above to return different metadata by adding/removing/changing fields in the ``return`` phrase. (Consult the :ref:`list of publications fields ` for ideas.) As a shorthand for commonly requested data, the DSL also supports a few :term:`fieldsets
` for each source, so that the name of a fieldset can be used instead of a long list of fields. For example, the ``extras`` fieldset includes additional fields we may not have seen before: .. code-block:: none search publications for "malaria" return publications [extras] Compare the results of the query above to those of the query below, which uses the smaller ``basics`` fieldset: .. code-block:: none search publications for "malaria" return publications[basics] The ``basics`` fieldset is the default used when no fields or fieldsets are specified in the results phrase, such that ``return publications`` returns the exact same results as ``return publications[basics]``. Try deleting ``:basics`` from the query above to see for yourself. For a complete list of supported fields and fieldsets for each source, see the :ref:`supported data ` page of the documentation. Calculate aggregations ++++++++++++++++++++++ Sometimes we are not interested in data about each individual document, but rather in aggregate data about the set of documents matched by the return phrase. For example, perhaps you're interested in publications about malaria, but don't want to see the publications themselves; instead, you want to see the number of publications that appeared in each year. You can ask the DSL for that information with a ``return`` phrase like this: .. code-block:: none search publications for "malaria" return year When given this query, the DSL will find all publications matching the ``search`` phrase, then look at the ``year`` of each of those publications to tally up the ``count`` of matching publications for each year; the results will then present the ``years`` (not the ``publications`` themselves) with the highest publication counts. We may also be interested in the number of malaria publications supported by different funding organizations, which we could determine with a query like: .. code-block:: none search publications for "malaria" return year return funders We can calculate such aggregations via an operation called "faceting", which is supported for certain :term:`fields ` of the given :term:`source`; such fields are called :term:`facets `. Consult the tables of :ref:`supported source fields ` to learn which fields are facets. .. exercise:: Change the query above to aggregate over another ``publications`` facet. (Consult the :ref:`list of publications fields ` for ideas.) Specify entity fields ~~~~~~~~~~~~~~~~~~~~~ Since ``funders`` are an :term:`entity`, the Dimensions database stores some metadata about each of these funders. Just as we could specify which :term:`fields ` we wanted to see for each of the ``publications`` we returned earlier, we can specify which fields (or :term:`fieldset`) we'd like to see for each of the ``funders`` in the same way. Perhaps we're only interested in each funder's name: .. code-block:: none search publications for "malaria" return funders[name] Compare this to the results of the previous query, where we saw additional data like the ``acronym`` and ``country_name`` of each funder. .. exercise:: Play with the query above to add fields to the ``funders`` results (consult the :ref:`list of funding organization fields `). .. note:: Specifying fields is only possible for :term:`entity` fields of a given :term:`source`, such as ``funders`` of ``publications``. With a non-entity :term:`field`, such as ``years``, we cannot specify fields; a year has no name or other metadata. Which fields of a given source are entities/non-entities can be seen in the field tables for the supported :ref:`data-sources`, and for each of the :ref:`supported-entities` there is also a list of available fields/fieldsets. Use different aggregation indicators ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ So far we've been interested in the number of publications each funder supported, but perhaps we're interested in knowing which funders supported the most-cited publications. In that case, instead of just tallying the count of publications per funder, we could use a different aggregation indicator, e.g. take the average of the Relative Citation Ratio (RCR) score for each publication supported by a given funder. We can tell the DSL to do this by specifying ``rcr_avg`` as the aggregation indicator: .. code-block:: none search publications for "malaria" return funders aggregate rcr_avg As you can see in the results of the query above, now we see an ``rcr_avg`` score for each funder, but we still see the ``count`` in the results (this is always included in aggregation results), and the funders are still ranked by ``count``. This is because ``count`` is the default sort indicator for any aggregations; if we want to sort by a different indicator, we just need to add a ``sort by`` phrase: .. code-block:: none search publications for "malaria" return funders aggregate rcr_avg sort by rcr_avg Compare the results of the query above to those of the previous query; as you can see, now the funders appear ranked by their average RCR score, not publication count. Consult the list of indicators for each :ref:`supported source ` to find out which indicators(s) can be used. .. exercise:: Try updating the query above to aggregate & sort the funders by the median Altmetric Attention Score of the publications they supported. Limit the number of results +++++++++++++++++++++++++++ In all of the results we've seen so far, whether we're returning ``publications`` documents or an aggregation over ``years`` or ``funders``, we've seen the top 20 entries for each. That is because the DSL is configured to return 20 results by default, and for those to be the first 20 results. You can customize how many results are returned by setting a ``limit``, as follows: .. code-block:: none search publications for "malaria" return funders limit 5 .. exercise:: Try it yourself by changing the ``limit`` in the query above. How high can it go? Return multiple results +++++++++++++++++++++++ Instead of writing two separate queries like these that use the same ``search`` phrase and are asking for an aggregation over the results, like ``search publications for "malaria" return years`` and ``search publications for "malaria" return funders``, we can combine them using multiple ``return`` phrases, like so: .. code-block:: none search publications for "malaria" return years return funders If you wish to *also* get data about the publications themselves, you can ask for that in a separate ``return`` phrase, like this: .. code-block:: none search publications for "malaria" return year return funders return publications And if you want to use a different aggregation indicator or limit for funders, but not years, you'll need to use separate return phrases for each, e.g.: .. code-block:: none search publications for "malaria" return year return funders aggregate rcr_avg limit 5 return publications One way of looking at this is that we are using multiple ``return`` phrases to get multiple different "views" over a single set of data (the documents matching the ``search`` phrase). .. warning:: Returning field both as a facet and a field might result in incomplete results. For example, this query: ``search publications return publications[researchers] return researchers[extras]`` would fail to retrieve ``extras`` fieldset for the ``researchers`` facet, and instead only the ``basic`` fieldset would be returned. In this case, it might be better to split the query into two and return facet separately.