Overview

fmu-dataio is a specialized Python library designed to manage data in Fast Model Update (FMU) workflows.

Key Features

  • Data compliance: Ensures data exports meet FMU standards, including file and folder conventions.

  • Rich metadata: Attaches detailed metadata to exported data, enhancing its usability for various consumers.

  • Versatile usage: Supports all stages of FMU workflows, including Ert FORWARD_MODELs, from within RMS scripts, and pre-/post-processing tasks.

Purpose

The primary goal of fmu-dataio is to simplify the data export process and provide context to the data generated by FMU workflows. This context is important for making the data understandable and usable beyond the FMU context. Since filenames alone cannot capture all necessary context, fmu-dataio generates and attaches rich metadata to exported files.

While fmu-dataio simplifies data management with exports, it also lays the groundwork for further enhancements, such as:

  • Efficient result management in Sumo

  • Simple methods to retrieve data already stored in Sumo

  • Centralized post-processing services

  • Automated data pipelines to the Reservoir Experience Platform

  • A new cloud-only version of Webviz

  • Data availability to other current and future Equinor services, applications, etc.

Installation

Within Equinor, fmu-dataio will already be available if you are using a stable version of Komodo. You can test its availability in Python:

import fmu.dataio

fmu-dataio is also available from within RMS Python, which for most users is the most common way to use it.

pip

The latest stable release of fmu-dataio can be installed via pip. It is recommended to do this in a Python virtual environment.

pip install fmu-dataio

If you want to use the in-development version, you can install directly from the GitHub git repository.

pip install git+https://github.com/equinor/fmu-dataio

Exporting data

There are two ways to export data using fmu-dataio.

  1. Simple exports, which produce standard results

  2. Custom exports, which produce custom results

Where and how these results (exported data) can be used differs inside and outside of the FMU context.

Simple exports, standard results

Simple exporters are functions defined in fmu-dataio that export standard results. A standard result is exported data that is in conformance with the FMU data standard. These functions are meant to be trivial in their usage, but are less flexible in the different ways data can be produced.

One example of this is that a simple export may mandate that tabular data is exported with a particular set of column headers. These headers may be strict in their number, ordering, capitalization, etc. This is an important way to make data legible between different models.

Using a simple export from fmu-dataio looks like this. This example is from a Python script in RMS, in an FMU workflow:

from fmu.dataio.export.rms import export_inplace_volumes

export_results = export_inplace_volumes(project, "Geogrid", "geogrid_volumes")

for result in export_results.items:
    print(f"Output volumes to {result.absolute_path}")

Important

This is the recommended way to export data.

Custom exports, custom results

It is also possible to export custom results using custom exports. These are exports that may be more flexible in the way that data can be exported. However, as they are not validated to conform to the FMU data standard, it is possible they cannot be used in different visualization and analysis tools both inside and outside of FMU.

A rough example looks like this:

from fmu.dataio import ExportData

df = create_data() # Some function that creates a Pandas dataframe
cfg = get_global_config() # The FMU global configuration

# ExportData can take many arguments. This is a simplified example.
exp = ExportData(
    config=cfg,
    content="volumes",
)
exp.export(df) # Exports the Pandas dataframe as a csv by default

Warning

Exporting custom results is not recommended over exporting standard results. Custom results may have local workflow use-cases but in general cannot be expected to be useful across different FMU and company solutions.

However, there may not yet be a simple exporter for every data type. In this case a custom export is your only option.

Exported files

When storing standard results to disk, fmu-dataio specifies a file location, name, and format according to the defined data standard. This may or may not follow existing conventions but offers consistency between models.

When storing custom results, a best effort is made to store the data file according to the current filename-oriented FMU data standard.

In either case, with a corresponding file name fmu-dataio will store a “hidden” metadata file describing different things about the data that has been exported.

# Standard result
share/results/tables/inplace_volumes/gridname.parquet
share/results/tables/inplace_volumes/.gridname.parquet.yml # <-- Metadata

# Custom result
share/results/tables/mytable.csv
share/results/tables/.mytable.csv.yml # <-- Metadata

Context on multiple levels are added to the metadata. First of all, references to Equinor master data is required, but also FMU-specific context such as which model was used to produce the data, which realization and iteration a specific file is produced in, and much more.

Note

Metadata are in general “data about the data”. For instance a pure surface export file (e.g. irap binary) will have information about data itself (like values in the map, and bounding box) but the file has no idea of which FMU run it belongs to, or which field it comes from, or what data contents it is representing.

Some metadata contents are static for the model, meaning that they will be the same for all runs of a specific model setup. Some metadata contents are static to the case, meaning that they will be the same for all data belonging to a specific case. Other metadata are object-specific, i.e. they will vary per item which is exported.

Next Steps

Before you can begin exporting data with fmu-dataio, some set up is required. This is covered in the Getting Started section.