.. _data-sources:

Data Sources
==============


This section provides information about each :term:`source` currently supported
by the DSL (version |version|) in its default configuration,
listing each :term:`field` available for that source with details
about the field value type, including whether it may be used as a :term:`facet`
and whether it is a multi-value field.
The page also lists the fields included in each :term:`fieldset` available
for a given source,
and each aggregation :term:`indicator` available for each source.

There are eight primary content types that can be queried via the API: 

.. toctree::
   :maxdepth: 1

   datasource-publications
   datasource-grants
   datasource-patents
   datasource-clinical_trials
   datasource-policy_documents
   datasource-datasets
   datasource-source_titles
   datasource-reports

Two additional sources consist of curated lists of disambiguated researchers and organizations. These objects are referenced in all of the content types above.  

.. toctree::
   :maxdepth: 1

   datasource-researchers
   datasource-organizations
   datasource-funder_groups
   datasource-research_org_groups

The figure below provides an overview of the main document sources and their respective cross links.

.. image:: _static/images/data-model-overview-1.png
    :width: 80%
    :align: center
    :alt: cross source links - document sources 

The figure below provides an overview of the main relatioships between document sources and researchers / organizations. 

.. image:: _static/images/data-model-overview-2.png
    :width: 80%
    :align: center
    :alt: cross source links - documents and researchers-organizations 

Finally, a number of auxiliary entities exists, which cannot be queried directly but are available as structured fields or facets attached to the primary sources. The next 




.. _data-entities:

Auxiliary Entities
-----------------------------

Auxiliary entities are subsidiary objects that cannot be queried directly, but are available as structured data attached to primary sources via fields or facets.


.. jinja:: config

    {% set Entities = config.docs_entities %}

    {% for entity in Entities %}
    {% if entity not in Sources %}
    .. |{{ entity }}| replace:: ``{{ entity }}``
    {% endif %}
    {% endfor %}

    .. _supported-entities:

    {% for entname, entity in Entities|dictsort|reverse %}

    {% set entity_name_nice = entname|replace("_", " ")|title %}

    .. _{{ entname }}:

    {{ entity_name_nice }}
    +++++++++++++++++++++++++++++++++++++++

    {{ entity_name_nice }} Literal Fields
    -------------------------------------
    .. list-table::
       :widths: auto
       :header-rows: 1

       {% set has_descriptions = entity.literals|rejectattr("description", "none")|list|length is greaterthan 0 %}

       * - Field
         - Literal Type
         - Multi?
         - Filter? Entity Filter?
         {% if has_descriptions %}
         - Description
         {% endif %}
       {% for fieldname, field in entity.literals|dictsort|reverse %}
       * - ``{{ fieldname }}``
         - :ref:`{{ field.documentation_type }} <supported-types>`
         - {% if field.is_multi %}**✓**{% endif %}
         - {% if field.is_filter %}**✓** {% if field.is_entity_filter %}**✓**{% endif %}{% endif %}
         {% if has_descriptions %}
         - {% if field.description is not none %}{{ field.description }}{% endif %} {% if field.long_description is not none %}[#{{entname}}_{{fieldname}}_long_desc]_{% endif %}
         {% endif %}
       {% endfor %}


    {% for fieldname, field in entity.literals|dictsort %}
    {% if field.long_description is not none %}
    .. [#{{entname}}_{{fieldname}}_long_desc] {{ field.long_description }}
    {% endif %}
    {% endfor %}


    {{ entity_name_nice }} Fieldsets
    --------------------------------
    .. list-table::
       :widths: auto
       :header-rows: 1

       * - Fieldset
         - Fields
       {% for fieldset in entity.fieldset_names|sort|reverse %}
       {% if fieldset != "all" %}
       * - ``{{ fieldset }}``
         - {% for field in entity.fieldset_fields(fieldset)|sort %}``{{ field }}``
           {% endfor %}
       {% endif %}
       {% endfor %}

    {% endfor %}




.. _supported-types:

Literal Field Types
-----------------------------


This section provides information about the most basic field types currently supported
by the DSL (version |version|).

.. data-literals:


.. jinja:: config

    {% set Types = config.literal_types %}


    .. list-table::
       :widths: auto
       :header-rows: 1

       * - Field type
         - Description
       * - ``date``
         - String that represents a precise date. Format is ``"YYYY-MM-DD"``, if ``MM`` or ``DD`` are omitted, they are considered to be ``01`` automatically.
       * - ``float``
         - Floating decimal point number
       * - ``integer``
         - Whole number
       * - ``string``
         - Short piece of text a few characters/words long, e.g. journal issue
       * - ``json``
         - Nested unstructured data


    .. end




.. _describe:

Metadata API
---------------------------------------

The DSL exposes programmatically metadata, such as supported sources and entities, along with their fields, facets, fieldsets, metrics and search fields.

This is available via the ``describe`` command. This can be used in the following ways:

- ``describe`` - returns list of all sources and entities
- ``describe version`` - returns information about the DSL version and release
- ``describe source <source_name>`` - returns information about fields, fieldsets, metrics and search_fields of a source
- ``describe entity <entity_name>`` - returns information about fields and fieldsets of an entity
- ``describe schema`` - returns information about all sources and all entities at once
- ``describe service <service_name>`` - returns information about a service. Supported services: `grid`


For example, the following query will retrieve metadata about all the fields relevant to the ``publications`` source.

.. code-block:: none

   describe source publications

The metadata is returned as JSON.