Data model

Data model

hermes’ internal data model acts like a contract between hermes and plugins. It is based on JSON-LD (JSON Linked Data), and the public API simplifies interaction with the data model through Python code.

Output of the different hermes subcommands consequently are valid JSON-LD files (deposit plugins are an exception) that are cached in subdirectories of the .hermes/ directory that is created in the root of the project directory (see following diagram).

.hermes
│  audit.log
├──curate
│  └──result
│     ├──codemeta.json
│     ├──context.json
│     └──expanded.json
├──deposit
│  └──invenio <- for every plugin one folder
│     ├──deposit.json <- not necessarily JSON-LD
│     └──result.json <- not necessarily JSON-LD
├──harvest
│  └──cff <- for every plugin one folder
│     ├──codemeta.json
|     ├──context.json
│     └──expanded.json
└──process
   └──result
      ├──codemeta.json
      ├──context.json
      └──expanded.json

The cache should only be interacted with via the hermes libraries.

Depending on whether you develop a plugin for hermes, or you develop hermes itself, you need to know either some, or quite a few things about JSON-LD.

The following sections provide documentation of the data model. They aim to help you get started with hermes plugin and core development, even if you have no previous experience with JSON-LD.

The data model for plugin developers

If you develop a plugin for hermes, you will only need to work with three Python classes and the public API they provide: hermes.model.api.SoftwareMetadata, hermes.model.types.ld_dict.ld_dict and hermes.model.types.ld_list.ld_list

To work with those classes, it is necessary that you know some things about JSON-LD.

JSON-LD for plugin developers

Attention

Work in progress.

Working with the hermes data model in plugins

Goal
Understand how plugins access the hermes data model and interact with it.

hermes aims to hide as much of the data model as possible behind a public API to avoid that plugin developers have to deal with some of the more complex features of JSON-LD.

You can extend hermes with plugins for all five different commands: harvest, process, curate, deposit, postprocess.

The commands differ in how they work with instances of the data model.

  • harvest plugins create a single new model instance and return it.

  • process plugins to do not interact with any model instance and only supply Hermes with strategies to do so.

  • curate plugins are passed a single existing model instance (the output of process), and return a single model instance.

  • deposit plugins are passed a single existing model instance (the output of curate), but currently do not return a model instance but instead JSON in their own format.

  • postprocess plugins currently do not interact with any model instance but only with the JSON data of the associated deposit plugin

Important

Plugins access the data model exclusively through the API provided by the three classes SoftwareMetadata, ld_dict and ld_list.

The following sections show how those classes work (together).

Creating a data model instance

Model instances are primarily created in harvest plugins, but may also be created in other plugins to map existing data into.

To create a new model instance, initialize SoftwareMetadata:

Initializing a default data model instance
from hermes.model import SoftwareMetadata

data = SoftwareMetadata()

The __init__() method takes two arguments data and extra_vocabs (in this positional order).

If the SoftwareMetadata object is initialized without a value for data it is empty. Passing a value to the argument data initialized the model instance with passed data. This data must be a native python object that resembles valid JSON-LD data:

Initializing a model instance with data
from hermes.model import SoftwareMetadata

value = {
    "schema:name": "My software",
    "schema:description": [{"@value": "This is software."}],
    "http://schema.org/author": {"@type": "schema:Person", "schema:email": "test@example.com"}
}

data = SoftwareMetadata(value)

If the object is initialized without a value for extra_vocabs the default context (see JSON-LD for plugin developers) is used. This means that, you can only use terms from the schemas included in the default context to describe software metadata. That means terms from CodeMeta can be used without a prefix, i.e. only readme, while terms from Schema.org can be used with the prefix schema, e.g. schema:copyrightNotice. Of course the absolute IRI can be used too, e.g. https://codemeta.github.io/terms/readme and https://schema.org/copyrightNotice.

You can also use other linked data vocabularies. To do this, you need to identify them with a prefix and register them with the data model by passing it extra_vocabs as a dict mapping prefixes to URLs where the vocabularies are provided as JSON-LD:

Injecting additional schemas
from hermes.model import SoftwareMetadata

# Contents served at https://example.com/schema.jsonld:
# {
#    "@context":
#    {
#       "name": "https://schema.org/name"
#    }
# }

data = SoftwareMetadata(extra_vocabs={"foo": "https://example.com/schema.jsonld"})

data["foo:name"] = ...

Adding data

Once you have an instance of ld_dict or its subclass SoftwareMetadata (cf. for ld_list below), you can add data to it, i.e., metadata that describes software:

Setting data values in ld_dicts
from datetime import datetime
from hermes.model import SoftwareMetadata

data = SoftwareMetadata()
data["name"] = "My Research Software"  # A simple "Text"-type value (int, float and bool are supported too)
data["author"] = {"name": "Shakespeare"}  # An object value that uses terms available in the defined context
data["schema:description"] = ["software for research", "more descriptions", "Why so many descriptions?"]  # lists of values
data["schema:dateCreated"] = datetime.now()  # A datetime object (time and date are also supported)
# → Simplified representation of data:
#{
#    "name": ["My Research Software"],
#    "author": [{"name": "Shakespeare"}],
#    "schema:description": ["software for research", "more descriptions", "Why so many descriptions?"],
#    "schema:dateCreated": ["{Iso format string}"]
#}
# Cf. "Accessing data" below

Other methods to add data are:

Setting data values in ld_dicts (advanced)
from hermes.model import SoftwareMetadata

data = SoftwareMetadata()
data.emplace("name")  # similar to data["name"]=[] but does not overwrite
data.setdefault("schema:description", "my description")  # similar to data["schema:description"]="my description" but does not overwrite data and returns the value after the operation
data.update({"name": "foo", "schema:dateCreated": []})  # just like dict.update()

Adding data to ld_list can be done like that:

Setting data values in ld_lists
from datetime import datetime

data_list.append("bits")  # appending
data_list.extend(["apples", "paper"])  # extending
data_list[1] = "foo"  # replacing value
data_list[1:3] = ["bytes", "silicon"]  # replacing values
# → Simplified representation of data_list: ["bits", "bytes", "silicon"]
# Cf. "Accessing data" below

Accessing data

You need to be able to access data in the data model instance to add, edit or remove data. Data can be accessed by using term strings, similar to how values in Python dicts are accessed by keys.

Important

When you access data from a data model instance or a ld_dict, it will always be returned in a list-like object (of class ld_list)!

The reason for providing data in list-like objects is that expanded JSON-LD treats all property values as arrays (source). Even if you add “single value” data to a hermes data model instance via the API, the underlying JSON-LD model will treat it as an array, i.e., returns it as an object of ld_list:

Accessing data from ld_dicts
# → Simplified representation of data: {"name": ["My Research Software"]}
# All following statements return an ld_list whose simplified representation is [ "My Research Software" ]
data["name"]
data.get("name")
data.setdefault("name", value)  # where data["name"]=value would be evoked, if data hadn't contained a value for "name", and then data["name"] returned anyways

ld_dict also implements keys() and compact_keys() that return a iterator-like view on the expanded or compacted keys respectively. items() works like items() from dict but returns the keys in their expanded version.

Accessing data from a ld_list can be done similar but returns ld_dicts, ld_lists, strings, int, bool, datetime (or time or date) objects. Therefore, you access data in the same way you would access data from a Python list:

  1. You access single values using indices and slices, e.g., data["name"][0].

  2. You can use a list-like API to interact with data objects, e.g. for name in data["name"]: ....

Interacting with data

The following longer example shows different ways that you can interact with SoftwareMetadata objects and the data API.

Building the data model
from hermes.model import SoftwareMetadata

# Create the model object with the default context
data = SoftwareMetadata()

# Let's create author metadata for our software!
# Below each line of code, the value of `data["author"]` is given.

data["author"] = {"name": "Shakespeare"}
# [{'name': ['Shakespeare']}]

data["author"].append({"name": "Hamilton"})
# [{'name': ['Shakespeare']}, {'name': ['Hamilton']}]

data["author"][0]["email"] = "shakespeare@example.net"
# [{'name': ['Shakespeare'], 'email': ['shakespeare@example.net']}, {'name': ['Hamilton']}]

data["author"][1].emplace("email") # instead of testing whether a key is contained in the ld_dict and setting or appending, just emplace the key
# [{'name': ['Shakespeare'], 'email': ['shakespeare@example.net']}, {'name': ['Hamilton'], 'email': []}]

data["author"][1]["email"].append("hamilton@example.net")
# [{'name': ['Shakespeare'], 'email': ['shakespeare@example.net']}, {'name': ['Hamilton'], 'email': ['hamilton@example.net']}]

data["author"][1]["email"].extend(["hamilton@example.org", "hamilton@example.com"])
# [
#   {'name': ['Shakespeare'], 'email': ['shakespeare@example.net']},
#   {'name': ['Hamilton'], 'email': ['hamilton@example.net', 'hamilton@example.org', 'hamilton@example.com']}
# ]

The example continues to show how to iterate through data and test whether a value is contained.

for-loop, containment check
for i, author in enumerate(data["author"], start=1):
    if any(name in author["name"][0] for name in ["Shakespeare", "Hamilton"]):
        print(f"Author {i} has expected name.")
    else:
        raise ValueError("Unexpected author name found!", author["name"][0])

# Mock output:
# $> Author 1 has expected name.
# $> Author 2 has expected name.
Value check
for email in data["author"][0]["email"]:
    if email.endswith(".edu"):
        print("Shakespeare has an email address at an educational institution.")
        break
else:
    print("Cannot confirm affiliation with educational institution for Shakespeare.")

# Mock output
# $> Cannot confirm affiliation with educational institution for Shakespeare.
Value check and list comprehension
if all(["hamilton" in email for email in data["author"][1]["email"]]):
    print("Author has only emails with their name in it.")

# Mock output
# $> Author has only emails with their name in it.

The example continues to show how to assert data values.

The API class hides the internal model objects. Therefore you can use many different values (compacted, expanded, some mix of them, etc.) that will be treated as the same object.

Containment assertion
try:
    assert (
        {'name': [{'@value': 'Shakespeare'}], 'http://schema.org/email': ['shakespeare@example.net']}
        in
        data["author"]
    )
    print("The author was found!")
except AssertionError:
    print("The author could not be found.")
    raise

# Mock output
# $> The author was found!
#
#
# Internal Model from data["author"]:
# {'@list': [
#     {
#         'http://schema.org/name': [{'@value': 'Shakespeare'}],
#         'http://schema.org/email': [{'@value': 'shakespeare@example.net'}]
#     },
#     {
#         'http://schema.org/name': [{'@value': 'Hamilton'}],
#         'http://schema.org/email': [
#                 {'@value': 'hamilton@example.net'}, {'@value': 'hamilton@example.org'}, {'@value': 'hamilton@example.com'}
#         ]
#     }
# ]}

The classes also support complex equality checks that work just like the presented containment checks.

See Also