Shift to the middle of each interval

Description

After thickening all the values are either shifted to the first or the last value of their interval. This function creates a vector from x, with the values shifted to the (approximate) center of the interval. This can give a more accurate picture of the aggregated data when plotting.

Usage

center_interval(x, shift = c("up", "down"), interval = NULL)

Arguments

Details

The interval will be translated to number of days when x is of class Date, or number of seconds when x is of class POSIXt. For months and quarters this will be the average length of the interval. The translated units divided by two will be added by or subtracted from each value of x.

Value

Vector of the same class as x, with the values shifted to the (approximate) center.

Examples

library(dplyr)
library(ggplot2)
plot_set <- emergency %>%
  thicken("hour", "h") %>%
  count(h) %>%
  head(24)

ggplot(plot_set, aes(h, n)) + geom_col()

plot_set %>%
  mutate(h_center = center_interval(h)) %>%
  ggplot(aes(h_center, n)) + geom_col()

Retrieve the closest given weekday

Description

Find the closest instance of the requested weekday to min(x). Helper function for thicken with the interval "week", when the user desires the start day of the weeks to be different from Sundays.

Usage

closest_weekday(x, wday = 1, direction = c("down", "up"))

Arguments

Value

Object of class Date, reflecting the closest desired weekday to x.

Examples

closest_weekday(coffee$time_stamp)
closest_weekday(coffee$time_stamp, 5)
closest_weekday(coffee$time_stamp, 1, direction = "up")
closest_weekday(coffee$time_stamp, 5, direction = "up")

Coffee Data Set

Description

Made-up data set for demonstrating padr.

Usage

coffee

Format

A data frame with 4 rows and 2 variables:

time_stamp

YYYY-MM-DD HH:MM:SS

amount

Amount spent on coffee

Emergency Calls for Montgomery County, PA

Description

The emergency calls coming in at Montgomery County, PA since 2015-12-10. Data set was created at 2016-10-17 16:15:40 CEST from the API and contains events until 2016-10-17 09:47:03 EST. From the original set the columns desc and e are not included.

Usage

emergency

Format

A data frame with 120450 rows and 6 variables:

lat

Latitude from Google maps, based on the address

lng

Longitude from Google maps, based on the address

zip

Zipcode from Google, when possible

title

Title, emergency category

time_stamp

YYYY-MM-DD HH:MM:SS

twp

Township

Fill missing values by a function of the nonmissings

Description

For each specified column in x replace the missing values by a function of the nonmissing values.

Usage

fill_by_function(x, ..., fun = mean)

Arguments

Value

x with the altered columns.

Examples

library(dplyr) # for the pipe operator
x <- seq(as.Date('2016-01-01'), by = 'day', length.out = 366)
x <- x[sample(1:366, 200)] %>% sort
x_df <- data_frame(x  = x,
                   y1 = runif(200, 10, 20) %>% round,
                   y2 = runif(200, 1, 50) %>% round)
x_df %>% pad %>% fill_by_function(y1, y2)
x_df %>% pad %>% fill_by_function(y1, y2, fun = median)

Fill missing values by the most prevalent nonmissing value

Description

For each specified column in x replace the missing values by the most prevalent nonmissing value.

Usage

fill_by_prevalent(x, ...)

Arguments

Value

x with the altered columns.

Examples

library(dplyr) # for the pipe operator
x <- seq(as.Date('2016-01-01'), by = 'day', length.out = 366)
x <- x[sample(1:366, 200)] %>% sort
x_df <- data_frame(x  = x,
                  y1 = rep(letters[1:3], c(80, 70, 50)) %>% sample,
                  y2 = rep(letters[2:5], c(60, 80, 40, 20)) %>% sample)
x_df %>% pad %>% fill_by_prevalent(y1, y2)

Fill missing values by a single value

Description

Replace all missing values in the specified columns by the same value.

Usage

fill_by_value(x, ..., value = 0)

Arguments

Value

x with the altered columns.

Examples

library(dplyr) # for the pipe operator
x <- seq(as.Date('2016-01-01'), by = 'day', length.out = 366)
x <- x[sample(1:366, 200)] %>% sort
x_df <- data_frame(x  = x,
                   y1 = runif(200, 10, 20) %>% round,
                   y2 = runif(200, 1, 50) %>% round,
                   y3 = runif(200, 20, 40) %>% round,
                   y4 = sample(letters[1:5], 200, replace = TRUE))
x_padded <- x_df %>% pad
x_padded %>% fill_by_value(y1)
x_df %>% pad %>% fill_by_value(y1, y2, value = 42)

Make a period character vector

Description

After applying thicken all the observations of a period are mapped to a single time point. This function will convert a datetime variable to a character vector that reflects the period, instead of a single time point. strftime is used to format the start and the end of the interval.

Usage

format_interval(
  x,
  start_format = "%Y-%m-%d",
  end_format = start_format,
  sep = " ",
  end_offset = 0,
  units_to_last = NULL
)

Arguments

Details

The end of the periods will be determined by the next unique value in x. It does so without regarding the interval of x. If a specific interval is desired, thicken and / or pad should first be applied to create an equally spaced datetime variable.

Value

A character vector showing the interval.

Examples

library(dplyr)
library(ggplot2)
plot_set <- emergency %>%
  head(500) %>%
  thicken("hour", "h") %>%
  count(h)

# this will show the data on the full hour
ggplot(plot_set, aes(h, n)) + geom_col()

# adding a character to indicate the hours of the interval.
plot_set %>%
  mutate(h_int = format_interval(h, "%H", sep = "-"))

Get the interval of a datetime variable

Description

The interval is the highest datetime unit that can explain all instances of a variable of class Date, class POSIXct, or class POSIXct. This function will determine what the interval of the variable is.

Usage

get_interval(x)

Arguments

Details

See vignette("padr") for more information on intervals.

Value

A character string indicating the interval of x.

Examples

x_month <- seq(as.Date('2016-01-01'), as.Date('2016-05-01'), by = 'month')
get_interval(x_month)

x_sec <- seq(as.POSIXct('2016-01-01 00:00:00'), length.out = 100, by = 'sec')
get_interval(x_sec)
get_interval(x_sec[seq(0, length(x_sec), by = 5)])

Pad the datetime column of a data frame

Description

pad will fill the gaps in incomplete datetime variables, by figuring out what the interval of the data is and what instances are missing. It will insert a record for each of the missing time points. For all other variables in the data frame a missing value will be inserted at the padded rows.

Usage

pad(
  x,
  interval = NULL,
  start_val = NULL,
  end_val = NULL,
  by = NULL,
  group = NULL,
  break_above = 1
)

Arguments

Details

The interval of a datetime variable is the time unit at which the observations occur. The eight intervals in padr are from high to low year, quarter, month, week, day, hour, min, and sec. Since padr v.0.3.0 the interval is no longer limited to be of a single unit. (Intervals like 5 minutes, 6 hours, 10 days are possible). pad will figure out the interval of the input variable and the step size, and will fill the gaps for the instances that would be expected from the interval and step size, but are missing in the input data. Note that when start_val and/or end_val are specified, they are concatenated with the datetime variable before the interval is determined.

Rows with missing values in the datetime variables will be retained. However, they will be moved to the end of the returned data frame.

Value

The data frame x with the datetime variable padded. All non-grouping variables in the data frame will have missing values at the rows that are padded. The result will always be sorted on the datetime variable. If group is not NULL result is sorted on grouping variable(s) first, then on the datetime variable.

Examples

simple_df <- data.frame(day = as.Date(c('2016-04-01', '2016-04-03')),
                        some_value = c(3,4))
pad(simple_df)
pad(simple_df, interval = "day")

library(dplyr) # for the pipe operator
month <- seq(as.Date('2016-04-01'), as.Date('2017-04-01'),
              by = 'month')[c(1, 4, 5, 7, 9, 10, 13)]
month_df <- data.frame(month = month,
                       y = runif(length(month), 10, 20) %>% round)
# forward fill the padded values with tidyr's fill
month_df %>% pad %>% tidyr::fill(y)

# or fill all y with 0
month_df %>% pad %>% fill_by_value(y)

# padding a data.frame on group level
day_var <- seq(as.Date('2016-01-01'), length.out = 12, by = 'month')
x_df_grp <- data.frame(grp1 = rep(LETTERS[1:3], each =4),
                       grp2 = letters[1:2],
                       y    = runif(12, 10, 20) %>% round(0),
                       date = sample(day_var, 12, TRUE)) %>%
 arrange(grp1, grp2, date)

# pad by one grouping var
x_df_grp %>% pad(group = 'grp1')

# pad by two groups vars
x_df_grp %>% pad(group = c('grp1', 'grp2'), interval = "month")

# Using group argument the interval is determined over all the observations,
# ignoring the groups.
x <- data.frame(dt_var = as.Date(c("2017-01-01", "2017-03-01", "2017-05-01",
"2017-01-01", "2017-02-01", "2017-04-01")),
id = rep(1:2, each = 3), val = round(rnorm(6)))
pad(x, group = "id")
# applying pad with do, interval is determined individualle for each group
x %>% group_by(id) %>% do(pad(.))

Pad with a custom spanning

Description

Pad the datetime variable after thicken_cust is applied, using the same spanning.

Usage

pad_cust(x, spanned, by = NULL, group = NULL, drop_last_spanned = TRUE)

Arguments

Value

The data frame x with the datetime column padded.

Examples

library(dplyr)
# analysis of traffic accidents in traffic jam hours and other hours.
accidents <- emergency %>% filter(title == "Traffic: VEHICLE ACCIDENT -")
spanning <- span_time("20151210 16", "20161017 17", tz = "EST") %>%
  subset_span(list(hour = c(6, 9, 16, 19)))
thicken_cust(accidents, spanning, "period") %>%
  count(period) %>%
  pad_cust(spanning)

Pad the integer column of a data frame

Description

pad_int fills the gaps in incomplete integer variables. It will insert a record for each of the missing value. For all other variables in the data frame a missing value will be inserted at the padded rows.

Usage

pad_int(x, by, start_val = NULL, end_val = NULL, group = NULL, step = 1)

Arguments

Value

The data frame x with the specified variable padded. All non-grouping variables in the data frame will have missing values at the rows that are padded.

Examples

int_df <- data.frame(x = c(2005, 2007, 2008, 2011),
                     val = c(3, 2, 6, 3))
pad_int(int_df, 'x')
pad_int(int_df, 'x', start_val = 2006, end_val = 2013)

int_df2 <- data.frame(x = c(2005, 2015), val = c(3, 4))
pad_int(int_df2, 'x', step = 2)
pad_int(int_df2, 'x', step = 5)

int_df3 <- data.frame(x = c(2005, 2006, 2008, 2006, 2007, 2009),
                      g = rep(LETTERS[1:2], each = 3),
                      val = c(6, 6, 3, 5, 4, 3))
pad_int(int_df3, 'x', group = 'g')
pad_int(int_df3, 'x', group = 'g', start_val = 2005, end_val = 2009)

Span an equally spaced vector around a datetime variable

Description

Span a vector of specified interval around a variable of class Date, POSIXct, or POSIXlt..

Usage

span_around(x, interval, start_shift = NULL, end_shift = start_shift)

Arguments

Details

Note that use of the start_shift and end_shift arguments change the entire spanning when they are not in line with the interval. It is not checked for.

Value

A datetime vector, with the first observation smaller or equal than min(x) and the last observation larger or equal than max(x). Spaces between points are equal to interval.

Examples

span_around(coffee$time_stamp, "hour")
span_around(coffee$time_stamp, "hour", end_shift = "2 hour")
span_around(coffee$time_stamp, "2 day")
span_around(coffee$time_stamp, "2 day", start_shift = "2 day")
span_around(emergency$time_stamp, "week")
span_around(emergency$time_stamp, "2 month")

Wrapper around seq.Date.

Description

Quickly create a sequence of dates from minimal specifications.

Usage

span_date(from, to = NULL, len_out = NULL, by = NULL)

Arguments

Details

Minimal specification of dates, sets unspecified date parts to default values. These are 01 for both month and day.

In addition to from, either to or len_out must be specified. If by is not specified, span_date will set the interval to the highest of the specified date parts in either from or to. For example, if they are 2011 and 2015 it will be "year", if they are 2011 and 201501 it will be "month".

Value

An object of class Date.

Examples

# using "to" argument
span_date(2011, 2015)
span_date(201101, 201501)
span_date(2011, 2015, by = "month")
span_date(2011, 201501)
span_date(20111225, 2012)

# using "len_out" argument
span_date(2011, len_out = 4)
span_date(201101, len_out = 4)
span_date(20110101, len_out = 4)
span_date(20110101, len_out = 4, by = "month")

Wrapper around seq.POSIXct.

Description

Quickly create a sequence of datetimes from minimal specifications.

Usage

span_time(from, to = NULL, len_out = NULL, by = NULL, tz = "UTC")

Arguments

Details

Minimal specification of datetimes, sets unspecified date parts to default values. These are 01 for both month and day and 00 for hour, minute, and second.

In addition to from, either to or length must be specified. If the by is not specified, span_time will set the interval to the highest of the specified datetime parts in either from or to. For example, if they are "20160103 01" and "20160108 05" it will be "hour", if they are "2011" and "20110101 021823" it will be "second".

Value

An object of class POSIXct.

Examples

# using to
span_time(2011, 2013)
span_time("2011", "2013")
span_time(2011, 201301)
span_time(2011, 20130101)
span_time(2011, "20110101 0023")
span_time(2011, "20110101 002300")

# using len_out
span_time(2011, len_out = 3)
span_time("2011", len_out = 3)
span_time(2011, len_out = 10, by = "month")
span_time(2011, len_out = 10, by = "day")
span_time(2011, len_out = 10, by = "hour")
span_time("20110101 00", len_out = 10)
span_time("20110101 002300", len_out = 10)

Subset a spanned datetime vector

Description

Take a Date, POSIXct, or POSIXlt vector and subset it by a pattern of date and/or time parts.

Usage

subset_span(spanned, pattern_list)

Arguments

Details

For subsetting weekdays, they run from 0 (Sunday) to 6 (Saturday).

Value

Vector of the same class as spanned, containing all the data points in spanned that meets the requirements in pattern_list.

Examples

date_span <- span_date(20170701, len_out = 100)
subset_span(date_span, list(wday = 1:5))

time_span <- span_time("20170101 00", 201702)
subset_span(time_span, list(hour = 7:17))
subset_span(time_span, list(hour = c(10, 16), mday = seq(5, 30, 5)))

Add a variable of a higher interval to a data frame

Description

Take the datetime variable in a data frame and map this to a variable of a higher interval. The mapping is added to the data frame in a new variable.

Usage

thicken(
  x,
  interval,
  colname = NULL,
  rounding = c("down", "up"),
  by = NULL,
  start_val = NULL,
  drop = FALSE,
  ties_to_earlier = FALSE
)

Arguments

Details

When the datetime variable contains missing values, they are left in place in the dataframe. The added column with the new datetime variable, will have a missing values for these rows as well.

See vignette("padr") for more information on thicken. See vignette("padr_implementation") for detailed information on daylight savings time, different timezones, and the implementation of thicken.

Value

The data frame x with the variable added to it.

Examples

x_hour <- seq(lubridate::ymd_hms('20160302 000000'), by = 'hour',
              length.out = 200)
some_df <- data.frame(x_hour = x_hour)
thicken(some_df, 'week')
thicken(some_df, 'month')
thicken(some_df, 'day', start_val = lubridate::ymd_hms('20160301 120000'))

library(dplyr)
x_df <- data.frame(
  x = seq(lubridate::ymd(20130101), by = 'day', length.out = 1000) %>%
    sample(500),
  y = runif(500, 10, 50) %>% round) %>%
  arrange(x)

# get the max per month
x_df %>% thicken('month') %>% group_by(x_month) %>%
  summarise(y_max = max(y))

# get the average per week, but you want your week to start on Mondays
# instead of Sundays
x_df %>% thicken('week',
                 start_val = closest_weekday(x_df$x, 2)) %>%
  group_by(x_week) %>% summarise(y_avg = mean(y))

# rounding up instead of down
x <- data.frame(dt = lubridate::ymd_hms('20171021 160000',
                                        '20171021 163100'))
thicken(x, interval = "hour", rounding = "up")
thicken(x, interval = "hour", rounding = "up", ties_to_earlier = TRUE)

Thicken with a custom spanning

Description

Like thicken, it will find the datetime variable in x and add a variable of a higher periodicity to it. However, the variable to which to map the observation is provided by the user. This enables mapping to time points that are unequally spaced.

Usage

thicken_cust(x, spanned, colname, by = NULL, drop = FALSE)

Arguments

Details

Only rounding down is available for custom thickening.

Value

The data frame x with the variable added to it.

Examples

library(dplyr)
# analysis of traffic accidents in traffic jam hours and other hours.
accidents <- emergency %>% filter(title == "Traffic: VEHICLE ACCIDENT -")
spanning <- span_time("20151210 16", "20161017 17", tz = "EST") %>%
  subset_span(list(hour = c(6, 9, 16, 19)))
thicken_cust(accidents, spanning, "period") %>%
  count(period) %>%
  pad_cust(spanning)