Postdiction in the Distributed Setting

Trevor Petersen

Project Overview

Problem: This project was aimed at introducing postdiction, a lossy reduction method designed for databases, to the distributed setting. Postdiction can have significant computational overhead and slow execution times, expecially when compared to other lossless compression algorithms. This inspired the search

Approach: The main way this problem was approached was by introducing fitting methods to the postdiction process. Previously, postdiction would use all of the data given to it to train its models. This was changed into a two-phase approach: first a training phase then a fitting phase. The training phase would use a sample of the data to create the models, and the fitting phase would fit all new points into their model. One important consideration during both of these was ensuring that our program error threshold (defined by the user) was never violated. This threshold ensured that all values recovered via postdiction are within error range of their original value.

I tested three different fitting methods for this project:

• Method 1: Using a binary search heuristic to identify a likely candidate before testing the point on a model
• Method 2: Using interval trees to break down the space into squares that were entirely in the range of a models error threshold
• Method 3: Using an array implementation of Method 2 to speed up searches by having better cache/memory locality.

Results: The outcome was that Method 3 seemed like the fastest option for processing data and was able to outperform the speed of training. Additionally, fitting can occur on many different cores, devices, or machines, which can offer even greater speed ups.

Project Materials

• Final Presentation (PDF)
• Final Report (PDF)
• Source Code (.zip)