May All Data be Reproducible And Transparent (madrat) *

Description

Package provides a basic framework which should improve reproducibility and transparency in data processing. It provides functionality such as automatic meta data creation and management, rudimentary quality management, data caching, work-flow management an data aggregation.

Details

* The title is a wish not a promise. By no means we expect this package to deliver everything what is needed to achieve full reproducibility and transparency, but we believe that it supports efforts in this direction.

Author(s)

Maintainer: Jan Philipp Dietrich dietrich@pik-potsdam.de (ORCID) (Potsdam Institute for Climate Impact Research)

Authors:

• Lavinia Baumstark lavinia@pik-potsdam.de (Potsdam Institute for Climate Impact Research)

• Stephen Wirth wirth@pik-potsdam.de (Potsdam Institute for Climate Impact Research)

• Anastasis Giannousakis giannou@pik-potsdam.de

• Renato Rodrigues Renato.Rodrigues@pik-potsdam.de (Potsdam Institute for Climate Impact Research)

• Benjamin Leon Bodirsky bodirsky@pik-potsdam.de (Potsdam Institute for Climate Impact Research)

• Debbora Leip leip@pik-potsdam.de (Potsdam Institute for Climate Impact Research)

• Ulrich Kreidenweis kreidenweis@pik-potsdam.de

• David Klein dklein@pik-potsdam.de (Potsdam Institute for Climate Impact Research)

• Pascal Sauer pascal.sauer@pik-potsdam.de (Potsdam Institute for Climate Impact Research)

See Also

Useful links:

• https://github.com/pik-piam/madrat

• doi:10.5281/zenodo.1115490

• Report bugs at https://github.com/pik-piam/madrat/issues

addMapping

Description

Function whichs adds another mapping to the current list of extramappings in the madrat configuration (see setConfig) and stores the mapping in the mapping folder as well as output folder.

Usage

addMapping(filename, mapping = NULL)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

setConfig

Examples

## Not run: 
addMapping("regionmappingH12.csv")

## End(Not run)

Tool: cacheArgumentsHash

Description

Calculate hash from given function arguments for given call

Usage

cacheArgumentsHash(call, args = NULL)

Arguments

Value

A hash representing the given arguments hash for the given call. NULL, if no argument deviates from the default argument settings.

Author(s)

Jan Philipp Dietrich

See Also

cachePut, cacheName, getNonDefaultArguments

Examples

madrat:::cacheArgumentsHash("madrat:::readTau", args = list(subtype = "historical"))
madrat:::cacheArgumentsHash("madrat:::readTau", args = list(subtype = "paper"))
calls <- c(madrat:::readTau, madrat:::convertTau)
madrat:::cacheArgumentsHash(calls, args = list(subtype = "historical"))

cacheCleanup

Description

Delete files older than the specified number of days, based on file time metadata (per default atime = last access time).

Usage

cacheCleanup(
  daysThreshold,
  path = getConfig("cachefolder", verbose = FALSE),
  timeType = c("atime", "mtime", "ctime"),
  ask = TRUE,
  readlineFunction = readline
)

Arguments

Details

File time metadata is not available on all systems and the semantics are also system dependent, so please be careful and check that the correct files are deleted. This function will return a data.frame containing all files that would be deleted if the user answers 'n' to the question. If deleting files fails a warning is created.

Value

If the user answers 'n', a data.frame as returned by base::file.info, containing only files older than <daysThreshold> days.

cacheCopy

Description

Copy cache files which were used for a given preprocessing.

Usage

cacheCopy(file, target = NULL, filter = NULL)

Arguments

Value

A vector of cache files which match the given log information and filter.

Author(s)

Jan Philipp Dietrich

Tool: cacheGet

Description

Load fitting cache data (if available)

Usage

cacheGet(prefix, type, args = NULL, graph = NULL, ...)

Arguments

Value

cached data, if cache is available, otherwise NULL

Author(s)

Jan Philipp Dietrich

See Also

cachePut, cacheName

Examples

madrat:::cacheGet("calc", "TauTotal", packages = "madrat")

Tool: cacheName

Description

Load fitting cache data (if available)

Usage

cacheName(
  prefix,
  type,
  args = NULL,
  graph = NULL,
  mode = "put",
  packages = getConfig("packages"),
  globalenv = getConfig("globalenv")
)

Arguments

Value

Name of fitting cache file, if available, otherwise NULL

Note

setConfig(forcecache=TRUE) strongly affects the behavior of cacheName. In read model it will also return cache names with deviating hashes if no fitting cache file is found (in that case it will just return the newest one). In write mode the hash in the name will be left out since due to cache forcing it cannot be guaranteed that the cache file agrees with the state represented by the hash.

Author(s)

Jan Philipp Dietrich

See Also

cachePut, cacheName

Examples

madrat:::cacheName("calc", "TauTotal")

Tool: cachePut

Description

Save data to cache

Usage

cachePut(x, prefix, type, args = NULL, graph = NULL, ...)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

cachePut, cacheName

Examples

## Not run: 
example <- 1
madrat:::cachePut(example, "calc", "Example", packages = "madrat")

## End(Not run)

calcOutput

Description

Calculate a specific output for which a calculation function exists. The function is a wrapper for specific functions designed for the different possible output types.

Usage

calcOutput(
  type,
  aggregate = TRUE,
  file = NULL,
  years = NULL,
  round = NULL,
  signif = NULL,
  supplementary = FALSE,
  append = FALSE,
  warnNA = TRUE,
  na_warning = NULL,
  try = FALSE,
  regionmapping = NULL,
  writeArgs = NULL,
  ...
)

Arguments

Value

magpie object with the requested output data either on country or on regional level depending on the choice of argument "aggregate" or a list of information if supplementary is set to TRUE.

Note

The underlying calc-functions are required to provide a list of information back to calcOutput. Following list entries should be provided:

• x - the data itself as magclass object

• weight - a weight for the spatial aggregation

• unit - unit of the provided data

• description - a short description of the data

• note (optional) - additional notes related to the data

• class (optional | default = "magpie") - Class of the returned object. If set to something other than "magpie" most functionality, such as aggregation or unit tests will not be available and is switched off!

• isocountries (optional | default = TRUE (mostly) or FALSE (if global)) - a boolean indicating whether data is in iso countries or not (the latter will deactivate several features such as aggregation)

• mixed_aggregation (optional | default = FALSE) - boolean which allows for mixed aggregation (weighted mean mixed with summations). If set to TRUE weight columns filled with NA will lead to summation, otherwise they will trigger an error.

• min (optional) - Minimum value which can appear in the data. If provided calcOutput will check whether there are any values below the given threshold and warn in this case

• max (optional) - Maximum value which can appear in the data. If provided calcOutput will check whether there are any values above the given threshold and warn in this case

• structure.spatial (optional) - regular expression describing the name structure of all names in the spatial dimension (e.g. "^[A-Z]\{3\}$"). Names will be checked against this regular expression and disagreements will be reported via a warning.

• structure.temporal (optional) - regular expression describing the name structure of all names in the temporal dimension (e.g. "^y[0-9]\{4\}$"). Names will be checked against this regular expression and disagreements will be reported via a warning.

• structure.data (optional) - regular expression describing the name structure of all names in the data dimension (e.g. "^[a-z]*\\\\.[a-z]*$"). Names will be checked against this regular expression and disagreements will be reported via a warning.

• aggregationFunction (optional | default = toolAggregate) - Function to be used to aggregate data from country to regions. The function must have the argument x for the data itself and rel for the relation mapping between countries and regions and must return the data as magpie object in the spatial resolution as defined in rel.

• aggregationArguments (optional) - List of additional, named arguments to be supplied to the aggregation function. In addition to the arguments set here, the function will be supplied with the arguments x, rel and if provided/deviating from the default also weight and mixed_aggregation.

• putInPUC (optional) boolean which decides whether this calculation should be added to a puc file which contains non-aggregated data and can be used to later on aggregate the data to resolutions of own choice. If not set calcOutput will try to determine automatically, whether a file is being required for the puc file or not, but in more complex cases (e.g. if calculations below top-level have to be run as well) this setting can be used to manually tweak the puc file list. CAUTION: Incorrect settings will cause corrupt puc files, so use this setting with extreme care and only if necessary.

• cache (optional) boolean which decides whether a cache file should be written (if caching is active) or not. Default setting is TRUE. This can be for instance useful, if the calculation itself is quick, but the corresponding file sizes are huge. Or if the caching for the given data type does not support storage in RDS format. CAUTION: Deactivating caching for a data set which should be part of a PUC file will corrupt the PUC file. Use with care.

Author(s)

Jan Philipp Dietrich

See Also

setConfig, calcTauTotal,

Examples

## Not run: 

a <- calcOutput(type = "TauTotal")

## End(Not run)

Calculate total tau

Description

This function prepares total tau values for use. As the source data already provides all required information this function purely removes not required data and moves xref values to the weighting object which is required for aggregation.

Usage

calcTauTotal(source = "paper")

Arguments

Value

Total tau data and corresponding weights as a list of two MAgPIE objects

Author(s)

Jan Philipp Dietrich

See Also

calcOutput, readTau, convertTau

Examples

## Not run: 
calcOutput("TauTotal")

## End(Not run)

cleanComment

Description

Helper function to clean a comment from additional metadata information

Usage

cleanComment(
  x,
  remove = c("unit", "description", "comment", "origin", "creation date", "note")
)

Arguments

Author(s)

Jan Philipp Dietrich

Examples

x <- maxample("animal")
getComment(x) <- c("unit: bla", "comment: hallo", "blub: ble")
madrat:::cleanComment(x)

compareData

Description

Compares the content of two data archives and looks for similarities and differences

Usage

compareData(x, y, tolerance = 10^-5, yearLim = NULL)

Arguments

Author(s)

Jan Philipp Dietrich, Florian Humpenoeder

See Also

setConfig, calcTauTotal,

Compare a madrat function's output with and without your changes

Description

With 'compareMadratOutputs' you can easily compare the output of a madrat function (read, calc, ...) with and without your changes. First, run 'compareMadratOutputs' without your changes, so a '.rds' file with the original output is saved. Then apply your changes and run 'compareMadratOutputs' again to compare the new output to the original output.

Usage

compareMadratOutputs(package, functionName, subtypes, overwriteOld = FALSE)

Arguments

Details

If there are differences a '<functionName>-new.rds' containing the new output is saved for closer inspection. All files are created in the current working directory.

Value

Invisibly the result of 'waldo::compare' or 'all.equal' if a comparison was made, otherwise a named list of the outputs for each subtype.

Author(s)

Pascal Sauer

Examples

## Not run: 
# save original output to readTau-old.rds
compareMadratOutputs("madrat", "readTau", c("paper", "historical"))

# now apply your changes to madrat:::readTau, reinstall madrat, restart the R session

# compare new output to original output from readTau-old.rds
compareMadratOutputs("madrat", "readTau", c("paper", "historical"))

## End(Not run)

Convert Tau

Description

Convert landuse intensity data (tau) to data on ISO country level.

Usage

convertTau(x)

Arguments

Value

Tau data and weights as MAgPIE object aggregated to country level

Author(s)

Jan Philipp Dietrich

downloadSource

Description

Download a source. The function is a wrapper for specific functions designed for the different possible source types.

Usage

downloadSource(type, subtype = NULL, overwrite = FALSE, numberOfTries = 300)

Arguments

Note

The underlying download-functions are required to provide a list of information back to downloadSource. Following list entries should be provided:

• url - full path to the file that should be downloaded

• title - title of the data source

• author - author(s) of the data set

• license - license of the data set. Put unknown if not specified.

• description - description of the data source

• unit - unit(s) of the data

• doi (optional) - a DOI URL to the data source

• version (optional) - version number of the data set

• release_date (optional) - release date of the data set

• reference (optional) - A reference for the data set (e.g. a paper, if the data was derived from it)

This user-provided data is enriched by automatically derived metadata:

• call - Information about the used madrat function call to download the data will check whether there are any values below the given threshold and warn in this case

• accessibility - A measure of quality for the accessibility of the data. Currently it distinguished between iron (manual access), silver (automatic access via URL) and gold (automatic access via DOI).

Besides the names above (user-provided and automatically derived) it is possible to add custom metadata entries by extending the return list with additional, named entries.

Author(s)

Jan Philipp Dietrich, David Klein, Pascal Sauer

See Also

setConfig, readSource

Examples

## Not run: 
a <- downloadSource("Tau", subtype = "historical")

## End(Not run)

findBottlenecks

Description

Analyzes a log from a retrieveData run, extracts runtime information for all called functions and identifies most critical bottlenecks.

Usage

findBottlenecks(file, unit = "min", cumulative = TRUE)

Arguments

Value

A data.frame sorted by net runtime showing for the different data processing functions their total runtime "time" (including the execution of all sub-functions) and net runtime "net" (excluding the runtime of sub-functions) and their share of total runtime.

Author(s)

Jan Philipp Dietrich

Tool: fingerprint

Description

Function which creates a unique fingerprint for a madrat function based on the code of the function itself, other madrat functions which are called by this function and of all source folders involved in the process. The fingerprint can serve as an indication whether the workflow for the given function has been most likely changed, or not. If all involved source folders and the code of all involved functions remains the same, also the fingerprint will stay the same, otherwise it will change. Hence, it can be to figure out whether a cache file can be used for further calculations, or whether the calculation should be redone.

Usage

fingerprint(name, details = FALSE, graph = NULL, ...)

Arguments

Value

A fingerprint (hash) of all provided sources, or "fingerprintError"

Note

For a better performance only the first 300 bytes of each file and the corresponding file size is hashed. As the fingerprint function only takes madrat-based functions into account (e.g. read-functions or calc-functions), but does ignore all other functions there might be cases where calculations actually changed, but the fingerprint is still the same. In a similar fashion it is possible that the fingerprint changes even though the workflow stayed the same (as the dependencies are sometimes overestimated).

Author(s)

Jan Philipp Dietrich, Pascal Sauer

See Also

readSource

Examples

madrat:::fingerprint("toolGetMapping", package = "madrat")

fullExample

Description

Example for class of fullX functions. Can be used as template for a new function or for testing the basic functionality

Usage

fullEXAMPLE(rev = 0, dev = "", extra = "Example argument")

Arguments

Author(s)

Jan Philipp Dietrich

See Also

readSource,getCalculations,calcOutput,setConfig

Examples

## Not run: 
retrieveData("example", rev = "2.1.2", dev = "test", regionmapping = "regionmappingH12.csv")

## End(Not run)

functionCallString

Description

Create a string representation for a function call. If the resulting string is longer than getConfig("maxLengthLogMessage") arguments are printed as passed (e.g. as variable name instead of the evaluated content of that variable), if that is still too long it is cropped.

Usage

functionCallString(functionName, argumentValues)

Arguments

Value

A string representing the given function call

Author(s)

Pascal Sauer

getCalculations

Description

This function can be used to retrieve a list of currently available sources and outputs (based on the availability of corresponding conversion functions in the loaded data data processing packages.)

Usage

getCalculations(
  prefix = "calc",
  packages = getConfig("packages"),
  globalenv = getConfig("globalenv")
)

Arguments

Value

A data frame containing all currently available outputs of all loaded data processing packages including its name, its function call and its package origin.

Author(s)

Jan Philipp Dietrich

See Also

readSource, setConfig

Examples

print(getCalculations())
print(getCalculations("read"))

getCode

Description

Extract function code from madrat-style functions in specified packages

Usage

getCode(
  packages = installedMadratUniverse(),
  globalenv = getConfig("globalenv")
)

Arguments

Value

A named vector with condensed function code

Author(s)

Jan Philipp Dietrich

See Also

getMadratGraph

getConfig

Description

This function returns the madrat config which is currently loaded. If no configuration has been loaded so far the configuration will be initialized with default settings or system settings (if available).

Usage

getConfig(option = NULL, raw = FALSE, verbose = TRUE, print = FALSE)

Arguments

Value

A config list with all settings currently set for the madrat package

Note

getConfig is primarily designed to make the overall madrat configuration available to system tools of the madrat framework. There are only a few exceptions for which configuration settings are also readable from within a download-, read-, convert-, correct-, calc- or full-function. These exceptions are the setting "debug" (which can be used to add additional debug messages when active), the "tmpfolder" which can be used to temporarily store data and the setting "hash" (which can only be accessed from within a full function and can there be used to apply the identical hash algorithm for other calculations in which hashing is being used). Besides that "regionmapping" and "extramappings" can also be read from within calc- and full-functions but their use is at least for the calc-functions discouraged as it either might lead to incorrect caching behavior, or - if implemented correctly - lead to significant slow-downs of overall calculations. All other settings are currently still accessible but trigger a warning that this option will soon be removed. So, please make sure that your code runs without reading these options! As a background note: Read access to these settings will be restricted as they otherwise would allow access to code elements or data in a form which is violating the overall madrat logic and thereby can lead to erroneous results.

Author(s)

Jan Philipp Dietrich

See Also

setConfig, initializeConfig

getDependencies

Description

Returns information about dependencies of a madrat-based calc- read- or full-function.

Usage

getDependencies(
  name,
  direction = "in",
  graph = NULL,
  type = NULL,
  self = FALSE,
  ...
)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

getCalculations, getMadratGraph, getMadratInfo

getFlags

Description

Support function which extracts flags from code. Flags are string literals in a function body, for example '"!# @pucArguments extra"'.

Usage

getFlags(code)

Arguments

Value

A list of found flag entries

Author(s)

Jan Philipp Dietrich

See Also

getCode

getFromComment

Description

Helper function extract a metadata comment

Usage

getFromComment(x, name)

Arguments

Author(s)

Jan Philipp Dietrich

Examples

x <- as.magpie(1)
getComment(x) <- c(" description: example description", " unit: kg")
getFromComment(x, "unit")
getFromComment(x, "description")

get official ISO list

Description

Function which returns the ISO list which is used as default for the input data preparation. It contains the countries to which all source data has to be up- or downscaled to.

Usage

getISOlist(type = "all", threshold = 1)

Arguments

Value

vector of default ISO country codes.

Note

Please always use this function instead of directly referring to the data object as the format in this data list might change in the future!

Author(s)

Jan Philipp Dietrich

See Also

getSources, getCalculations

Examples

head(getISOlist())
head(getISOlist("dispensable"))

getLinkFunction

Description

Returns a function that creates a symlink, hardlink, junction, or copy of files and directories, depending on OS capabilities (usually symlinks are not supported on Windows).

Usage

getLinkFunction()

Value

A function with arguments "from" and "to" which should behave like file.symlink on all platforms.

Author(s)

Pascal Sauer

getLocation

Description

Returns names of packages in which functions matching the description are found

Usage

getLocation(name, packages = installedMadratUniverse(), globalenv = TRUE)

Arguments

Value

vector of packages in which a function matching the description could be found

Author(s)

Jan Philipp Dietrich

See Also

getCalculations, getDependencies

getMadratGraph

Description

Function returns the madrat graph of all linkages of full, calc, and read functions of the given madrat based packages. Linkages to subfunctions of read functions (i.e. download, correct or convert functions) are not listed separately, but collectively referred to through the corresponding read function.

Usage

getMadratGraph(
  packages = installedMadratUniverse(),
  globalenv = getConfig("globalenv")
)

Arguments

Value

A data frame with 4 columns: from (source function), from_package (package the source function originates from), to (function which is using the source), to_package (package of the using function)

Author(s)

Jan Philipp Dietrich

See Also

getCalculations, getConfig

getMadratInfo

Description

Collects and returns detailed information about the currently loaded network of madrat functions.

Usage

getMadratInfo(graph = NULL, cutoff = 5, extended = FALSE, ...)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

getCalculations, getMadratGraph

getMadratMessage

Description

Read a madrat message from the madrat environment. The madrat environment behaves similar like global options, except that 1) messages will also be stored in cache files and restored when a cache file is being loaded and 2) messages are always stored in lists with messages split by function calls where the message was triggered.

Usage

getMadratMessage(name = NULL, fname = NULL)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

getMadratMessage

Examples

putMadratMessage("test", "This is a toast", fname = "readTau")
getMadratMessage("test", fname = "calcTauTotal")

getMainfolder

Description

Functions checks for a global setting of the mainfolder (either by setting the environment variable "MADRAT_MAINFOLDER" or by setting the R option with the same name). If none of these is available the user will be asked for a directory. If this is not provided a temporary folder will be used.

Usage

getMainfolder(verbose = TRUE, .testmode = FALSE)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

initializeConfig, getConfig, setConfig

getNonDefaultArguments

Description

Given a function and an argument list, identify which arguments are different from their default.

Usage

getNonDefaultArguments(call, args = NULL)

Arguments

Value

A subset of args that is used by the function/s and is different from default values.

Author(s)

Jan Philipp Dietrich

See Also

cacheArgumentsHash, toolstartmessage

Examples

madrat:::getNonDefaultArguments("madrat:::readTau", args = list(subtype = "historical"))
madrat:::getNonDefaultArguments("madrat:::readTau", args = list(subtype = "paper"))
calls <- c(madrat:::readTau, madrat:::convertTau)
madrat:::getNonDefaultArguments(calls, args = list(subtype = "historical"))

getSourceFolder

Description

Return the path to source data files for the given type and subtype. This applies redirections, see redirectSource for more details.

Usage

getSourceFolder(type, subtype)

Arguments

Value

Path to source data files

Author(s)

Pascal Sauer

getSources

Description

These functions can be used to retrieve a list of currently available sources and outputs (based on the availability of corresponding conversion functions in the loaded data data processing packages.)

Usage

getSources(
  name = NULL,
  type = NULL,
  packages = getConfig("packages"),
  globalenv = getConfig("globalenv")
)

Arguments

Value

A vector or data.frame containing all corresponding sources

Note

Please be aware that these functions only check the availability of corresponding functions of the package, not whether the functions will properly work.

Author(s)

Jan Philipp Dietrich

See Also

readSource, setConfig

Examples

print(getSources())

initializeConfig

Description

Checks whether configuration already has been set. If not, it will be initialized with default settings or (if available) system settings. All madrat folders (see setConfig for documentation which folders are available) will be set to the system environment variables MADRAT_SOURCEFOLDER, MADRAT_CACHEFOLDER, etc. if they exist, NA otherwise. NA means subfolders of the mainfolder are used.

Usage

initializeConfig(verbose = TRUE)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

getMainfolder, getConfig, setConfig

installedMadratUniverse

Description

Returns a name vector of installed packages which supposedly belong to the madrat universe. They are currently derived as the union of

• all packages registered under getConfig("packages"),

• all packages with a name starting with "mr" or "ms" (as the usual indicator for madrat-packages and madrat-support-packages), and

• all packages having madrat as either a Depends or Imports dependency.

Usage

installedMadratUniverse()

Value

A name vector of installed packages which supposedly belong to the madrat universe

Author(s)

Jan Philipp Dietrich

See Also

setConfig

Examples

## Not run: 
installedMadratUniverse()

## End(Not run)

isWrapperActive

Description

Support functions which checks whether a given wrapper function is currently in-use or not or which locally activate or deactivate a wrapper (setting will be automatically resetted when a function finishes).

Usage

isWrapperActive(name)

setWrapperActive(name)

setWrapperInactive(name)

Arguments

Functions

• setWrapperActive(): set wrapper activity status to on

• setWrapperInactive(): set wrapper activity status to off

Author(s)

Jan Philipp Dietrich

madTempDir

Description

returns a temporary directory as a subfolder of the tempfolder set in getConfig("tmpfolder").

Usage

madTempDir()

Value

path to the temp folder

Author(s)

Jan Philipp Dietrich

See Also

getConfig

Examples

## Not run: 
madrat:::madTempDir()

## End(Not run)

madapply

Description

This function is defunct and will be completely removed soon.

Usage

madapply(...)

Arguments

Author(s)

Jan Philipp Dietrich

madlapply

Description

This function is defunct and will be completely removed soon.

Usage

madlapply(...)

Arguments

Author(s)

Jan Philipp Dietrich

madratAttach / madratDetach

Description

Attaches the madrat functions of a package to the currently active madrat universe or detaches it again from it.

Usage

madratAttach(package)

madratDetach(package)

Arguments

Functions

• madratDetach(): detach package from madrat universe

Author(s)

Jan Philipp Dietrich

See Also

getConfig, setConfig

Examples

## Not run: 
madratAttach("madrat")

## End(Not run)

metadataGFZ

Description

Function to extract metadata information of a data set hosted at GFZ dataservices (https://dataservices.gfz-potsdam.de/portal/).

Usage

metadataGFZ(doi)

Arguments

Value

a list with entries "license", "citation", "authors" and "year"

Author(s)

Jan Philipp Dietrich

See Also

toolstartmessage, vcat

Examples

## Not run: 
metadataGFZ("10.5880/pik.2019.004")

## End(Not run)

prepComment

Description

Helper function to properly format a metadata comment entry

Usage

prepComment(x, name, warning = NULL)

Arguments

Author(s)

Jan Philipp Dietrich

Examples

 madrat:::prepComment("example comment", "example")

prepExtendedComment

Description

Helper function condense metadata information into an extended comment entry

Usage

prepExtendedComment(x, type, functionCallString, warn = TRUE)

Arguments

Author(s)

Jan Philipp Dietrich, Pascal Sauer

prepFunctionName

Description

Function to prepare a function call for a given type and prefix

Usage

prepFunctionName(type, prefix = "calc", ignore = NULL, error_on_missing = TRUE)

Arguments

Value

A function call as character to the specified function with corresponding package as attribute

Author(s)

Jan Philipp Dietrich

See Also

readSource, setConfig

Examples

print(madrat:::prepFunctionName("Tau", "read"))
print(madrat:::prepFunctionName("TauTotal", "calc"))
print(madrat:::prepFunctionName("EXAMPLE", "full"))

pucAggregate

Description

Function which takes a puc-file ("portable unaggregated collection") as created via retrieveData and computes the corresponding aggregated collection with the provided arguments (e.g. the provided region mapping). The resulting tgz-file containing the collection will be put to the madrat outputfolder as defined in getConfig("outputfolder").

Usage

pucAggregate(
  puc,
  regionmapping = getConfig("regionmapping"),
  ...,
  renv = TRUE,
  strict = FALSE
)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

retrieveData,localConfig

Examples

## Not run: 
pucAggregate("rev1_example.puc", regionmapping = "regionmappingH12.csv")

## End(Not run)

putMadratMessage

Description

Store a madrat message in the madrat environment. The madrat environment behaves similar like global options, except that 1) messages will also be stored in cache files and restored when a cache file is being loaded and 2) messages are always stored in lists with messages split by function calls where the message was triggered.

Usage

putMadratMessage(name, value, fname = -1, add = FALSE)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

putMadratMessage

Examples

putMadratMessage("test", "This is a toast", fname = "readTau")
getMadratMessage("test", fname = "calcTauTotal")

readSource

Description

Read in a source file and convert it to a MAgPIE object. The function is a wrapper for specific functions designed for the different possible source types.

Usage

readSource(
  type,
  subtype = NULL,
  subset = NULL,
  convert = TRUE,
  supplementary = FALSE
)

Arguments

Value

The read-in data, usually a magpie object. If supplementary is TRUE a list including the data and metadata is returned instead. The temporal and data dimensionality should match the source data. The spatial dimension should either match the source data or, if the convert argument is set to TRUE, should be on ISO code country level.

Note

If a magpie object is returned magclass::clean_magpie is run and if convert = TRUE ISO code country level is checked.

Author(s)

Jan Philipp Dietrich, Anastasis Giannousakis, Lavinia Baumstark, Pascal Sauer

See Also

setConfig, downloadSource, readTau #' @note The underlying read-functions can return a magpie object or a list of information (preferred) back to readSource. In list format the object should have the following structure:

• x - the data itself as magclass object

• unit (optional) - unit of the provided data

• description (otional) - a short description of the data

• note (optional) - additional notes related to the data

• class (optional | default = "magpie") - Class of the returned object. If set to something other than "magpie" most functionality will not be available and is switched off!

• cache (optional) boolean which decides whether a cache file should be written (if caching is active) or not. Default setting is TRUE. This can be for instance useful, if the calculation itself is quick, but the corresponding file sizes are huge. Or if the caching for the given data type does not support storage in RDS format. CAUTION: Deactivating caching for a data set which should be part of a PUC file will corrupt the PUC file. Use with care.

Examples

## Not run: 
a <- readSource("Tau", "paper")

## End(Not run)

Read Tau

Description

Read-in landuse intensity data (tau) following the methodology published in Dietrich J.P., Schmitz C., Mueller C., Fader M., Lotze-Campen H., Popp A., Measuring agricultural land-use intensity - A global analysis using a model-assisted approach, Ecological Modelling, Volume 232, 10 May 2012, Pages 109-118, ISSN 0304-3800, 10.1016/j.ecolmodel.2012.03.002.

Usage

readTau(subtype = "paper")

Arguments

Value

Tau data and weights as MAgPIE object in original resolution

Author(s)

Jan Philipp Dietrich

See Also

readSource

Examples

## Not run: 
a <- readSource("Tau")

## End(Not run)

redirect

Description

Redirect a given dataset type to a different source folder. The redirection is local, so it will be reset when the current function call returns. See example for more details.

Usage

redirect(type, target, linkOthers = TRUE, local = TRUE)

Arguments

Value

Invisibly, the source folder that is now used for the given type

Author(s)

Pascal Sauer

Examples

## Not run: 
f <- function() {
  redirect("Tau", target = "~/TauExperiment")
  # the following call will change directory
  # into ~/TauExperiment instead of <getConfig("sourcefolder")>/Tau
  readSource("Tau")
}
f()
# Tau is only redirected in the local environment of f,
# so it will use the usual source folder here
readSource("Tau")

## End(Not run)

redirectSource

Description

redirectSource will call a source specific redirect function if it exists (called e.g. redirectTau), in which case the arguments are passed on to that function. If such a function is not available redirect is called.

Usage

redirectSource(type, target, ..., linkOthers = TRUE, local = TRUE)

Arguments

Value

The result of the specific redirect function or redirect.

Author(s)

Pascal Sauer

redirectTau

Description

redirectTau will be called by redirectSource when type = "Tau". Redirects the Tau source folder to the target folder.

Usage

redirectTau(target, ...)

Arguments

Author(s)

Pascal Sauer

Examples

## Not run: 
redirectSource("Tau", "a/different/tau-source-folder")
a <- readSource("Tau", "paper")

## End(Not run)

Tool: regionscode

Description

Given a regionmapping (mapping between ISO countries and regions) the function calculates a regionscode which is basically the md5sum of a reduced form of the mapping. The regionscode is unique for each regionmapping and can be used to clearly identify a given regionmapping. In addition several checks are performed to make sure that the given input is a proper regionmapping

Usage

regionscode(mapping = NULL, label = FALSE, strict = TRUE)

Arguments

Value

A md5-based regionscode which describes the given mapping or, if label=TRUE and a corresponding label is available, the label belonging to the regionscode

Author(s)

Jan Philipp Dietrich

See Also

toolCodeLabels, fingerprint, digest

Examples

file <- system.file("extdata", "regionmappingH12.csv", package = "madrat")
regionscode(file)

resetMadratMessages

Description

Delete stored madrat messages from the madrat environment. The madrat environment behaves similar like global options, except that 1) messages will also be stored in cache files and restored when a cache file is being loaded and 2) messages are always stored in lists with messages split by function calls where the message was triggered.

Usage

resetMadratMessages(name = NULL, fname = NULL)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

putMadratMessage, getMadratMessage

Examples

putMadratMessage("test", "This is a toast", fname = "readTau")
getMadratMessage("test", fname = "calcTauTotal")
resetMadratMessages("test")

retrieveData

Description

Function to retrieve a predefined collection of calculations for a specific regionmapping.

Usage

retrieveData(
  model,
  rev = 0,
  dev = "",
  cachetype = "def",
  puc = identical(dev, ""),
  strict = FALSE,
  renv = TRUE,
  ...
)

Arguments

Value

Invisibly, the path to the newly created tgz archive.

Note

The underlying full-functions can optionally provide a list of information back to retrieveData. Following list entries are currently supported:

• tag (optional) - additional name tag which will be included in the file name of the aggregated collection (resulting tgz-file). This can be useful to highlight information in the file name which otherwise would not be visible.

• pucTag (optional) - identical purpose as tag but for the resulting unaggregated collections (puc-files).

Author(s)

Jan Philipp Dietrich, Lavinia Baumstark

See Also

calcOutput,setConfig

Examples

## Not run: 
retrieveData("example", rev = "2.1.1", dev = "test", regionmapping = "regionmappingH12.csv")

## End(Not run)

robustOrder, robustSort

Description

robustOrder: A wrapper around base::order that always uses the locale independent method = "radix". If the argument x is a character vector it is converted to utf8 first. robustSort: A convenience function using order to sort a vector using radix sort. The resulting vector will have the same encoding as the input although internally character vectors are converted to utf8 before ordering.

Usage

robustOrder(..., na.last = TRUE, decreasing = FALSE, method = "radix")

Arguments

Author(s)

Pascal Sauer

See Also

order

setConfig

Description

This function manipulates the current madrat configuration. In general, NULL means that the argument remains as it is whereas all other inputs will overwrite the current setting. For values which can be reset to NULL (currently only "extramappings") you can achieve a reset by setting the value to "".

Usage

setConfig(
  regionmapping = NULL,
  extramappings = NULL,
  packages = NULL,
  globalenv = NULL,
  enablecache = NULL,
  verbosity = NULL,
  mainfolder = NULL,
  sourcefolder = NULL,
  cachefolder = NULL,
  mappingfolder = NULL,
  outputfolder = NULL,
  pucfolder = NULL,
  tmpfolder = NULL,
  nolabels = NULL,
  forcecache = NULL,
  ignorecache = NULL,
  cachecompression = NULL,
  hash = NULL,
  diagnostics = NULL,
  debug = NULL,
  maxLengthLogMessage = NULL,
  redirections = NULL,
  .cfgchecks = TRUE,
  .verbose = TRUE,
  .local = FALSE
)

localConfig(...)

Arguments

Functions

• localConfig(): A wrapper for setConfig(..., .local = TRUE)

Note

setConfig must only be used before the data processing is started and changes in the configuration from within a download-, read-, correct-, convert-, calc-, or full-function are not allowed! Only allowed configuration update is to add another extramapping via addMapping. Currently the use of setConfig within any of these functions will trigger a warning, which is planned to be converted into an error message in one of the next package updates!

Author(s)

Jan Philipp Dietrich

See Also

getConfig, getISOlist

Examples

## Not run: 
setConfig(forcecache = c("readSSPall", "convertSSPall"))

## End(Not run)

toolAggregate

Description

(Dis-)aggregates a magclass object from one resolution to another based on a relation matrix or mapping

Usage

toolAggregate(
  x,
  rel,
  weight = NULL,
  from = NULL,
  to = NULL,
  dim = 1,
  wdim = NULL,
  partrel = FALSE,
  negative_weight = "warn",
  mixed_aggregation = FALSE,
  verbosity = 1,
  zeroWeight = "warn"
)

Arguments

Details

Basically toolAggregate is doing nothing more than a normal matrix multiplication which is taking into account the 3 dimensional structure of MAgPIE objects. So, you can provide any kind of relation matrix you would like. However, for easier usability it is also possible to provide weights for a weighted (dis-)aggregation as a MAgPIE object. In this case rel must be a 1-0-matrix or a mapping between both resolutions. The weight needs to be provided in the higher spatial aggregation, meaning for aggregation the spatial resolution of your input data and in the case of disaggregation the spatial resolution of your output data. The temporal and data dimension must be either identical to the resolution of the data set that should be (dis-)aggregated or 1. If the temporal and/or data dimension is 1 this means that the same transformation matrix is applied for all years and/or all data columns. In the case that a column should be just summed up instead of being calculated as a weighted average you either do not provide any weight (then all columns are just summed up) or your set this specific weighting column to NA and mixed_aggregation to TRUE.

Value

the aggregated data in magclass format

Author(s)

Jan Philipp Dietrich, Ulrich Kreidenweis

See Also

calcOutput

Examples

# create example mapping
p <- magclass::maxample("pop")
mapping <- data.frame(from = magclass::getItems(p, dim = 1.1),
                      region = rep(c("REG1", "REG2"), 5),
                      global = "GLO")
print(mapping)

# run aggregation
toolAggregate(p, mapping)
# weighted aggregation
toolAggregate(p, mapping, weight = p)
# combined aggregation across two columns
toolAggregate(p, mapping, to = "region+global")

Tool: CodeLabels

Description

This function replaces a hash code (e.g. regioncode) or another cryptic code with a human readable code via a given dictionary. This can be useful to make outputs better readable in cases where hash codes are already known to the user. If no entry exists in the dictionary the hash code is returned again.

Usage

toolCodeLabels(get = NULL, add = NULL)

Arguments

Value

A vector with either labels (if available) or hash codes (if no label was available).

Author(s)

Jan Philipp Dietrich

See Also

regionscode

Examples

toolCodeLabels("62eff8f7")

toolConditionalReplace

Description

Sets values (NA, negative, ..) to value replaceby

Usage

toolConditionalReplace(x, conditions, replaceby = 0)

Arguments

Value

return changed input data

Author(s)

Kristine Karstens

Tool: ConvertMapping

Description

Function which converts mapping files between formats

Usage

toolConvertMapping(name, format = "rds", type = NULL, where = "mappingfolder")

Arguments

Author(s)

Jan Philipp Dietrich

See Also

calcOutput, toolConvertMapping

toolCountry2isocode

Description

Function used to convert country names from the long name to the ISO 3166-1 alpha 3 country code

Usage

toolCountry2isocode(
  country,
  warn = TRUE,
  ignoreCountries = NULL,
  type = NULL,
  mapping = NULL
)

Arguments

Value

the ISO 3166-1 alpha 3 country code

Author(s)

Jan Philipp Dietrich, Anastasis Giannousakis

See Also

readSource,getSources

Examples

toolCountry2isocode("Germany")
toolCountry2isocode(c("Germany","Fantasyland"),mapping=c("Fantasyland"="BLA"))

Tool: CountryFill

Description

This function expects a MAgPIE object with ISO country codes in the spatial dimension. These ISO codes are compared with the official ISO code country list (stored as supplementary data in the madrat package). If there is an ISO code in the data but not in the official list this entry is removed, if an entry of the official list is missing in the data this entry is added and set to the value of the argument fill.

Usage

toolCountryFill(
  x,
  fill = NA,
  no_remove_warning = NULL,
  overwrite = FALSE,
  verbosity = 1,
  countrylist = NULL,
  ...
)

Arguments

Value

A MAgPIE object with spatial entries for each country of the official ISO code country list.

Author(s)

Jan Philipp Dietrich

Examples

library(magclass)
x <- new.magpie("DEU", 1994, "bla", 0)
y <- toolCountryFill(x, 99)

Tool: FillWithRegionAvg

Description

This function fills missing values for countries with the (weighted) average of the respective region. The average is computed separately for every timestep. Currently only inputs with one data dimension are allowed as inputs. (If the filling should be performed over multiple data dimensions, call this function multiple times and bind the results together with magclass::mbind.)

Usage

toolFillWithRegionAvg(
  x,
  valueToReplace = NA,
  weight = NULL,
  callToolCountryFill = FALSE,
  regionmapping = NULL,
  verbose = TRUE,
  warningThreshold = 0.5,
  noteThreshold = 1
)

Arguments

Details

toolFillWithRegionAvg can be used in conjunction with toolCountryFill() to first fill up the list of countries to the official ISO code country list, and then fill values with the regional average (see callToolCountryFill Option).

Value

A MAgPIE object with the missing values filled.

Author(s)

Bjoern Soergel, Lavinia Baumstark, Jan Philipp Dietrich

Examples

x <- magclass::new.magpie(cells_and_regions = c("A", "B", "C", "D"), years = c(2000, 2005),
  fill = c(1, NA, 3, 4, 5, 6, NA, 8))
rel <- data.frame(CountryCode = c("A", "B", "C", "D"), RegionCode = c("R1", "R1", "R1", "R2"))
xfilled <- toolFillWithRegionAvg(x, regionmapping = rel)

toolFillYears

Description

Inter- and extrapolates a historical dataset for a given time period.

Usage

toolFillYears(x, years)

Arguments

Value

MAgPIE object with completed time dimensionality.

Author(s)

Kristine Karstens

Tool: GetMapping

Description

Function which retrieves a mapping file

Usage

toolGetMapping(
  name,
  type = NULL,
  where = NULL,
  error.missing = TRUE,
  returnPathOnly = FALSE,
  activecalc = NULL
)

Arguments

Value

the mapping as a data frame

Author(s)

Jan Philipp Dietrich

See Also

calcOutput, toolConvertMapping

Examples

head(toolGetMapping("regionmappingH12.csv", where = "madrat"))

Tool: ISOhistorical

Description

This function expects a MAgPIE object with ISO country codes in the spatial dimension. For this MAgPIE object the time of transition is calculated and for each the historic time filled by using the mapping stored as supplementary data in the madrat package. If you want to use a different mapping please specify it in the argument mapping

Usage

toolISOhistorical(
  m,
  mapping = NULL,
  additional_mapping = NULL,
  overwrite = FALSE,
  additional_weight = NULL
)

Arguments

Value

A MAgPIE object with spatial entries for each country of the official ISO code country list. Historical time is filled up, old countries deleted

Author(s)

Lavinia Baumstark

Tool: ManualDownload

Description

Support tool for the creation of download functions in cases where a fully automated data download is not an option (e.g. due to a missing API). The function can be used to print a step-by-step guide for the user how to manually retrieve the data and then asks for a (local) path where the data can be copied from.

Usage

toolManualDownload(
  instructions,
  intro = "Data must be downloaded manually",
  request = "Enter full path to the downloaded data:"
)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

downloadSource

Examples

## Not run: 
toolManualDownload(c("Log into website ABC",
                     "Download the data set XYZ"))

## End(Not run)

Tool: NA replace

Description

Functions removes NAs, NaNs and infinite values in x and weight

Usage

toolNAreplace(x, weight = NULL, replaceby = 0, val.rm = NULL)

Arguments

Value

a list containing x and weight

Author(s)

Benjamin Bodirsky, Jan Philipp Dietrich

See Also

calcOutput

toolOrderCells

Description

reorder numbered spatial units (cells, clusters) by number. Function will return the unmodified object, if the given subdimension does not exist or does not contain cell information.

Usage

toolOrderCells(x, dim = 1.2, na.rm = FALSE)

Arguments

Value

ordered data in magclass format

Author(s)

Kristine Karstens, Jan Philipp Dietrich

Tool: SplitSubtype

Description

This function can split a subtype string into smaller entities based on a given separator and check whether these entities exist in a reference list

Usage

toolSplitSubtype(subtype, components, sep = ":")

Arguments

Value

A named list with the different entities of the given subtype

Author(s)

Jan Philipp Dietrich

Examples

toolSplitSubtype("mymodel:myversion:myworld", list(model=c("mymodel","notmymodel"), 
                                                   version=c("myversion","42"), 
                                                   world="myworld"))

Tool: SubtypeSelect

Description

This function is a support function for the selection of a subtype in a readX function. In addition to the subtype selection it also performs some consistency checks.

Usage

toolSubtypeSelect(subtype, files)

Arguments

Value

The file name corresponding to the given subtype

Author(s)

Jan Philipp Dietrich

See Also

readSource

Examples

files <-  c(protection="protection.csv",
              production="production.csv",
              extent="forest_extent.csv")
toolSubtypeSelect("extent",files)

toolTimeAverage

Description

average over time given an averaging range. Only works for data with equidistant time steps!

Usage

toolTimeAverage(x, averaging_range = NULL, cut = TRUE)

Arguments

Value

the averaged data in magclass format

Author(s)

Kristine Karstens, Jan Philipp Dietrich

toolTimeSpline

Description

Smoothing a data set by replacing its values by its spline approximation using the given degrees of freedom.

Usage

toolTimeSpline(x, dof = NULL)

Arguments

Value

approximated data in magclass format

Author(s)

Kristine Karstens, Felicitas Beier

toolXlargest

Description

Selects the countries with the highest values in a magpie object

Usage

toolXlargest(x, range = 1:20, years = NULL, elements = NULL, ...)

Arguments

Value

vector with ISO country codes

Author(s)

Benjamin Leon Bodirsky, Jan Philipp Dietrich

Examples

toolXlargest(magclass::maxample("pop"), range = 1:3)

Tool: End message

Description

This function writes a process end message and performs some diagnostics. It is always called after a corresponding call to toolstartmessage.

Usage

toolendmessage(startdata, level = NULL)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

toolstartmessage, vcat

Tool: Start message

Description

This function writes a process start message (what function was called with which arguments) and stores the current time, so the corresponding call to toolendmessage can calculate the elapsed time.

Usage

toolstartmessage(functionCallString, level = NULL)

Arguments

Value

A list containing diagnostic information required by toolendmessage.

Author(s)

Jan Philipp Dietrich, Pascal Sauer

See Also

toolendmessage, vcat

Examples

innerFunction <- function() {
  startinfo <- madrat:::toolstartmessage("innerFunction(argumentsToPrint = 123)", "+")
  vcat(1, "inner")
  madrat:::toolendmessage(startinfo, "-")
}
outerFunction <- function() {
  startinfo <- madrat:::toolstartmessage("outerFunction()", "+")
  vcat(1, "outer")
  innerFunction()
  madrat:::toolendmessage(startinfo, "-")
}
outerFunction()

Tool: Verbosity Cat

Description

Function which returns information based on the verbosity setting

Usage

vcat(
  verbosity,
  ...,
  level = NULL,
  fill = TRUE,
  show_prefix = TRUE,
  logOnly = FALSE
)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

readSource

Examples

## Not run: 
vcat(2, "Hello world!")

## End(Not run)

visualizeDependencies

Description

Creates a graphical visualization of dependencies between functions in the mr-universe.

Usage

visualizeDependencies(
  ...,
  direction = "both",
  order = 2,
  filter = NULL,
  packages = getConfig("packages"),
  filename = NULL
)

Arguments

Author(s)

Debbora Leip, Jan Philipp Dietrich

See Also

getDependencies, getMadratGraph, getMadratInfo

Tool: withMadratLogging

Description

Function will activate madrat logging facilities for all code provided to this function. This means that message, warning and stop calls will also report to the madrat log output

Usage

withMadratLogging(expr)

Arguments

Author(s)

Jan Philipp Dietrich

See Also

vcat

Examples

## Not run: 
madrat:::withMadratLogging(message("Hello world!"))

## End(Not run)