FAIR in Practice: Metadata, PIDs, and Data Formats
Module 4 of 5 — Open Science Training
Learning objectives¶
By the end of this notebook you will be able to:
Create a machine-readable metadata record using Python.
Explain the role of persistent identifiers (PIDs) and look up a real DOI.
Choose between open and proprietary file formats.
Write a basic
datapackage.json(Frictionless Data) descriptor.
Estimated time: 75–90 minutes
Level: Intermediate
Prerequisites: Modules 01–03
1. Persistent identifiers (PIDs)¶
A persistent identifier is a long-lasting reference to a digital object — it stays stable even if the object moves. The most common PIDs you will encounter in research:
| PID type | Used for | Example |
|---|---|---|
| DOI | Publications, datasets, software | 10.5281/zenodo.1234567 |
| ORCID | Researchers | 0000-0002-1825-0097 |
| ROR | Research organisations | https://ror.org/02mhbdp94 |
| ISNI | Institutions, people | 0000 0001 2193 314X |
| Handle | General digital objects | hdl:10.1000/182 |
Let’s look up a real DOI using the DataCite API:
import urllib.request
import json
# The original FAIR principles paper
doi = "10.1038/sdata.2016.18"
url = f"https://api.datacite.org/dois/{doi}"
try:
with urllib.request.urlopen(url, timeout=10) as response:
data = json.loads(response.read().decode())
attrs = data["data"]["attributes"]
print(f"Title: {attrs['titles'][0]['title']}")
print(f"Published: {attrs.get('publicationYear', 'N/A')}")
creators = attrs.get('creators', [])
print(f"Creators: {', '.join(c.get('name', '') for c in creators[:3])}{'...' if len(creators) > 3 else ''}")
print(f"DOI URL: https://doi.org/{doi}")
print(f"Publisher: {attrs.get('publisher', 'N/A')}")
licence = attrs.get('rightsList', [{}])
print(f"Licence: {licence[0].get('rights', 'not specified') if licence else 'not specified'}")
except Exception as e:
print(f"Could not reach DataCite API: {e}")
print("(This is normal if you are offline. The exercise still works.)")2. Choosing open file formats¶
Interoperability (the I in FAIR) requires open, non-proprietary formats wherever possible.
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import os
formats = [
# (format name, category, open?, long-term archival?)
("CSV", "Tabular", True, True),
("Excel .xlsx", "Tabular", False, False),
("JSON", "Structured", True, True),
("XML", "Structured", True, True),
("HDF5", "Scientific", True, True),
("NetCDF", "Scientific", True, True),
("SPSS .sav", "Statistical", False, False),
("STATA .dta", "Statistical", False, False),
("PDF/A", "Document", True, True),
("Word .docx", "Document", False, False),
("PNG/TIFF", "Image", True, True),
("PSD", "Image", False, False),
]
fig, ax = plt.subplots(figsize=(9, 5))
y = range(len(formats))
colors = ['#1D9E75' if f[2] else '#E24B4A' for f in formats]
ax.barh([f[0] for f in formats], [1]*len(formats), color=colors, height=0.6)
ax.set_xlim(0, 1.6)
ax.set_xticks([])
for i, f in enumerate(formats):
ax.text(1.05, i, f[1], va='center', fontsize=9, color='#444')
ax.text(0.5, i, 'Open' if f[2] else 'Proprietary', va='center',
ha='center', fontsize=9, color='white', fontweight='bold')
green_patch = mpatches.Patch(color='#1D9E75', label='Open / FAIR-friendly')
red_patch = mpatches.Patch(color='#E24B4A', label='Proprietary / avoid for archiving')
ax.legend(handles=[green_patch, red_patch], loc='lower right', fontsize=9)
ax.set_title('File format openness for FAIR data sharing', pad=12)
plt.tight_layout()
os.makedirs(out_dir, exist_ok=True)
plt.savefig('../img/formats.png', dpi=150, bbox_inches='tight')
plt.show()3. Creating a Frictionless Data Package descriptor¶
A Frictionless Data Package defines a lightweight, machine-readable datapackage.json file that describes a dataset. It is a practical, standards-based way to add FAIR metadata to any kind of data. Let’s use it for example to describe data in a *.csv file.
Let’s first create a small sample dataset in a csv format:
import pandas as pd
import json
import os
# Create a small toy dataset
os.makedirs('../data', exist_ok=True)
df = pd.DataFrame({
'site_id': ['S001', 'S002', 'S003', 'S004', 'S005'],
'latitude': [52.52, 48.85, 51.51, 41.90, 40.71],
'longitude': [13.40, 2.35, -0.13, 12.50, -74.01],
'temperature_change_pct': [23.4, 31.2, 18.9, 44.1, 27.5],
'measurement_date': ['2024-06-01', '2024-06-01', '2024-06-02', '2024-06-02', '2024-06-03'],
'collector_orcid': [
'0000-0000-0000-0001',
'0000-0000-0000-0001',
'0000-0000-0000-0002',
'0000-0000-0000-0002',
'0000-0000-0000-0001',
]
})
df.to_csv('../data/soil_temperature.csv', index=False)
print("Dataset created at ../data/soil_temperature.csv")
dfdatapackage = {
"name": "soil-temperature-example",
"title": "Example soil temperature change measurements",
"description": "Sample dataset of soil temperature changes for Open Science training purposes.",
"version": "1.0.0",
"created": "2025-05-01",
"licenses": [
{
"name": "CC-BY-4.0",
"title": "Creative Commons Attribution 4.0",
"path": "https://creativecommons.org/licenses/by/4.0/"
}
],
"contributors": [
{
"title": "Aleksandra Pawlik",
"role": "author",
"path": "https://orcid.org/0000-0001-8418-6735"
}
],
"keywords": ["soil moisture", "open science", "FAIR", "training"],
"resources": [
{
"name": "soil-temperature-example",
"path": "soil_temperature.csv",
"mediatype": "text/csv",
"schema": {
"fields": [
{"name": "site_id", "type": "string", "description": "Unique site identifier"},
{"name": "latitude", "type": "number", "description": "Decimal degrees, WGS84"},
{"name": "longitude", "type": "number", "description": "Decimal degrees, WGS84"},
{"name": "temperature_change_pct", "type": "number", "description": "Temperature change (%)"},
{"name": "measurement_date", "type": "date", "format": "%Y-%m-%d"},
{"name": "collector_orcid", "type": "string", "description": "ORCID of the data collector"}
],
"primaryKey": ["site_id", "measurement_date"]
}
}
]
}
with open('../data/datapackage.json', 'w') as f:
json.dump(datapackage, f, indent=2)
print("datapackage.json written to ../data/")
print(json.dumps(datapackage, indent=2))