How to transform your data¶

Outline:¶

This tutorial demonstrates how to use the following functions to transform a sample dataset in various ways. The tutorial covers the following steps:

1. Importing the brightwind library and loading some sample data

2. Using the scale_wind_speed() function

3. Using the average_data_by_period() function

4. Using the adjust_slope_offset() function

5. Using the offset_wind_direction() function

6. Using the offset_timestamps() function

7. Using the selective_avg() function

1. Importing the brightwind library and loading some sample data¶

[2]:

import brightwind as bw

The following commands will load the exisiting dataset and show the first few timestamps.

[3]:

# load existing sample dataset
data = bw.load_csv(bw.datasets.demo_data)

# show first few rows of dataframe
data.head(5)

[3]:

2. Using the scale_wind_speed() function¶

The following commands specify an arbitrary scale factor and apply this to one of the speed variables.

[4]:

# specify scale factor
scale_factor = 1.03

# apply scale factor to Spd80mN variable
Spd80mN_scaled = bw.scale_wind_speed(data.Spd80mN, scale_factor)

# print first 5 rows of result
Spd80mN_scaled.head(5)

[4]:

Timestamp
2016-01-09 15:30:00    8.62110
2016-01-09 15:40:00    8.49750
2016-01-09 17:00:00    7.88156
2016-01-09 17:10:00    7.60346
2016-01-09 17:20:00    8.21631
Name: Spd80mN, dtype: float64

Now we’ll repeat the same step, this time saving the resulting data in the existing dataframe. The new variable appears as the last column on the right hand side of the returned dataframe.

[5]:

# apply scale factor to Spd80mN variable, creating a new variable in the existing dataframe
data['Spd80mN_scaled'] = bw.scale_wind_speed(data.Spd80mN, scale_factor)

# show first few rows of updated dataframe, scaled wind speed is included in the last column
data.head(5)

[5]:

Finally, we can compare the mean values of the scaled and unscaled speed time series. The ratio of these mean values is shown to equal the intended scale factor.

[6]:

# print unscaled mean speed
print('The unscaled mean speed is: \t {} m/s'.format(round(data.Spd80mN.mean(),2)))

# print scaled mean speed
print('The scaled mean speed is: \t {} m/s'.format(round(data.Spd80mN_scaled.mean(),2)))

# calculate ratio of mean values
ratio = data.Spd80mN_scaled.mean()/data.Spd80mN.mean()

# print ratio of mean values
print('The ratio of these values is: \t {}'.format(round(ratio,2)))

The unscaled mean speed is:      7.5 m/s
The scaled mean speed is:        7.72 m/s
The ratio of these values is:    1.03

3. Using the average_data_by_period() function¶

This function can be used to apply a window average to data, using a window which can be specified as any number of minutes (‘min’), hours (‘H’), days (‘D’), weeks (‘W’). ‘1M’ and ‘1AS’ can also used for monthly and annual averages respectively. The output is created within a new series.

[7]:

# then derive monthly averages
data_monthly = bw.average_data_by_period(data.Spd80mN, period='1M')

# show monthly averaged data for first 5 months
data_monthly.head(5)

[7]:

Timestamp
2016-01-01    9.252377
2016-02-01    8.904382
2016-03-01    6.395166
2016-04-01    6.598875
2016-05-01    8.729657
Freq: MS, Name: Spd80mN, dtype: float64

4. Using the adjust_slope_offset() function¶

Now, let’s imagine that data from the northern anemometer at 80 m was logged with standard slope and offset values of 0.765 and 0.35, and we need to rescale this data considering slope and offset values from a calibration certificate which is specific to this anemometer. The following command can be used to do this, once again adding a new variable to the dataframe.

[8]:

# Apply slope and offset adjustments
data['Spd80mN_adj'] = bw.adjust_slope_offset(data.Spd80mN,0.765, 0.35, 0.7642, 0.352)

Now let’s view the raw and adjusted speed values side-by-side.

[9]:

# concatenate raw and adjusted variables
summary = data[['Spd80mN', 'Spd80mN_adj']]

# show first few values
summary.head(5)

[9]:

5. Using the offset_wind_direction() function¶

Similarly, the following function can be used to apply a scalar offset to direction data from a given wind vane.

[10]:

# apply directional offset
data['Dir38mS_adj'] = bw.offset_wind_direction(data.Dir38mS, -10)

# concatenate raw and adjusted variables
summary = data[['Dir38mS', 'Dir38mS_adj']]

# show first few values
summary.head(5)

[10]:

6. Using the offset_timestamps() function¶

This function can be used to apply an offset to the timestamps within a dataset. A few examples are provided below, first, let’s print the first timestamp of the raw data for comparison.

[11]:

# print first timestamp
print(data.index[0])

2016-01-09 15:30:00

Now we’ll add 1 hour and 30 minutes to the timestamps of the data and print the first timestamp once again.

[12]:

# add 90 minutes to timestamps
data = bw.offset_timestamps(data, '1.5H')

# print first timestamp
print(data.index[0])

2016-01-09 17:00:00

Negative offsets can also be applied.

[13]:

# subtract 2 hours from timestamps
data = bw.offset_timestamps(data, '-2H')

# print first timestamp
print(data.index[0])

2016-01-09 15:00:00

Offsets can also be applied in minutes (positive or negative).

[14]:

# add 7 minutes to timestamps
data = bw.offset_timestamps(data, '7min', overwrite=True)

# print first timestamp
print(data.index[0])

2016-01-09 15:07:00

Time offsets can also be restricted to a certain time period during the record using the following extra arguments. In this case we can see that the final adjustment has just been applied from the 2nd to the 4th timestamps.

[15]:

# subtract 7 minutes from timestamps between the datetime range below
data = bw.offset_timestamps(data, '-7min', date_from='2016-01-09 15:17:00', date_to='2016-01-09 16:47:00')

# show all data for first 5 timestamps
data.head(5)

[15]:

If the timestamp is adjustment is large enough to overlap the unadjusted data before/after the ‘date_from’/’date_to’, by default the function will not overwrite the unadjusted data. The argument ‘overwrite=True’ can be specified to change this default and overwrite the neighbouring timestamps.

Finally, note that the adjusted timestamps can be allocated to a completely new dataframe, leaving the input dataset unchanged.

[16]:

# create new dataset with 6H added to the original timestamps
data_adj = bw.offset_timestamps(data, '6H')

7. Using the selective_avg() function¶

This function can be used to create a time series of wind speed using data from two anemometers mounted at the same height, along with one wind vane. For each timestamp in the dataset, the function either averages the two wind speed values or only includes the upstream wind speed value if the other is in the wake of the mast. The wake is defined by winds approaching the mast from the opposite site of the boom within a directional sector of a given size (in this case 60 degrees).

[17]:

# derive selective average variable based on two 80 m anemeters and 78 m wind vane
data['sel_avg_80m'] = bw.selective_avg(data.Spd80mN, data.Spd80mS, wdir=data.Dir78mS, boom_dir_1=0, boom_dir_2=180, sector_width=60)

[18]:

# concatenate inputs and new output variable
summary = data[['Spd80mN', 'Spd80mS', 'Dir78mS','sel_avg_80m']]

# show the first 5 entries, none of which include waked anemometers
summary.head(5)

[18]: