# Overview

**fmu-dataio** is a specialized Python library designed to manage data in Fast
Model Update (FMU) workflows.

## Key Features

- Data compliance: Ensures data exports meet FMU standards, including
  file and folder conventions.
- Rich metadata: Attaches detailed metadata to exported data,
  enhancing its usability for various consumers.
- Versatile usage: Supports all stages of FMU workflows, including
  [Ert](https://github.com/equinor/ert) `FORWARD_MODEL`s, from within RMS
  scripts, and pre-/post-processing tasks.

## Purpose

The primary goal of fmu-dataio is to simplify the data export process and
provide context to the data generated by FMU workflows. This context is
important for making the data understandable and usable beyond the FMU
context. Since filenames alone cannot capture all necessary context,
fmu-dataio generates and attaches rich metadata to exported files.

While fmu-dataio simplifies data management with exports, it also lays the
groundwork for further enhancements, such as:

- Efficient result management in Sumo
- Simple methods to retrieve data already stored in Sumo
- Centralized post-processing services
- Automated data pipelines to the Reservoir Experience Platform
- A new cloud-only version of Webviz
- Data availability to other current and future Equinor services,
  applications, etc.

## Installation

Within Equinor, fmu-dataio will already be available if you are using a
stable version of Komodo. You can test its availability in Python:

```python
import fmu.dataio
```

fmu-dataio is also available from within RMS Python, which for most users is
the most common way to use it.

### pip

The latest stable release of fmu-dataio can be installed via pip. It is
recommended to do this in a [Python virtual
environment](https://docs.python.org/3/library/venv.html).

```shell
pip install fmu-dataio
```

If you want to use the in-development version, you can install directly from
the [GitHub git repository](https://github.com/equinor/fmu-dataio).

```shell
pip install git+https://github.com/equinor/fmu-dataio
```

## Exporting data

There are two ways to export data using fmu-dataio.

1. **Simple exports**, which produce **standard results**
2. **Custom exports**, which produce **custom results**

Where and how these results (exported data) can be used differs inside and
outside of the FMU context.

### Simple exports, standard results

**Simple exporters** are functions defined in fmu-dataio that export
**standard results**. A **standard result** is exported data that is in
conformance with the FMU data standard. These functions are meant to be
trivial in their usage, but are less flexible in the different ways data can
be produced.

One example of this is that a simple export may mandate that tabular data is
exported with a particular set of column headers. These headers may be strict
in their number, ordering, capitalization, etc. This is an important way to
make data legible between different models.

Using a simple export from fmu-dataio looks like this. This example is from a
Python script in RMS, in an FMU workflow:

```python
from fmu.dataio.export.rms import export_inplace_volumes

export_results = export_inplace_volumes(project, "Geogrid", "geogrid_volumes")

for result in export_results.items:
    print(f"Output volumes to {result.absolute_path}")
```

```{important}
This is the **recommended** way to export data.
```

### Custom exports, custom results

It is also possible to export **custom results** using custom exports. These
are exports that may be more flexible in the way that data can be exported.
However, as they are not validated to conform to the FMU data standard, it is
possible they cannot be used in different visualization and analysis tools
both inside and outside of FMU.

A rough example looks like this:

```python
from fmu.dataio import ExportData

df = create_data() # Some function that creates a Pandas dataframe
cfg = get_global_config() # The FMU global configuration

# ExportData can take many arguments. This is a simplified example.
exp = ExportData(
    config=cfg,
    content="volumes",
)
exp.export(df) # Exports the Pandas dataframe as a csv by default
```

```{warning}
Exporting custom results is **not recommended** over exporting standard
results. Custom results may have local workflow use-cases but in general
cannot be expected to be useful across different FMU and company
solutions.

However, there may not yet be a simple exporter for every data type. In this
case a custom export is your only option.
```

### Exported files

When storing **standard results** to disk, fmu-dataio specifies a file
location, name, and format according to the defined data standard. This may or
may not follow existing conventions but offers consistency between models.

When storing **custom results**, a best effort is made to store the data file
according to the current filename-oriented FMU data standard.

In either case, with a corresponding file name fmu-dataio will store a
"hidden" metadata file describing different things about the data that has
been exported.

```shell
# Standard result
share/results/tables/inplace_volumes/gridname.parquet
share/results/tables/inplace_volumes/.gridname.parquet.yml # <-- Metadata

# Custom result
share/results/tables/mytable.csv
share/results/tables/.mytable.csv.yml # <-- Metadata
```

Context on multiple levels are added to the metadata. First of all, references
to Equinor master data is required, but also FMU-specific context such as
which model was used to produce the data, which realization and iteration a
specific file is produced in, and much more.

```{note}
Metadata are in general "data about the data". For instance a pure surface
export file (e.g. irap binary) will have information about data itself (like
values in the map, and bounding box) but the file has no idea of which FMU run
it belongs to, or which field it comes from, or what data contents it is
representing.

Some metadata contents are _static for the model_, meaning that they will be
the same for all runs of a specific model setup. Some metadata contents are
*static to the case*, meaning that they will be the same for all data
belonging to a specific case. Other metadata are object-specific, i.e. they
will vary per item which is exported.
```

## Next Steps

Before you can begin exporting data with fmu-dataio, some set up is required.
This is covered in the [Getting Started](getting_started.md) section.