Spectre uses state of the art computer vision and machine learning to detect apple size, and then uses algorithms and statistical tools to estimate the size distribution of an apple bin in real time, all from one click on an iPhone or iPad.
Our team of passionate researchers, developers and engineers worked tirelessly for two years to produce Spectre and we are in awe of their talents and efforts. Early work involved US based warehouses, who provided amazing assistance with critical data collection. Initial Spectre pilots were carried out with growers across New Zealand, who provided fantastic feedback, assisting our team to improve Spectre. We continue to be humbled by the generosity shown by all of these businesses.
What is computer vision and machine learning?
Take a look at your hand and your eyes will see five fingers. Computer vision is where a camera or computer system can give you the same information, without you having to count. In order for computer vision to decipher that there are fingers in the photo, the system needs to be trained and taught what fingers look like. This is called machine learning.
Machine Learning requires a large amount of data to be input into a system for it to learn what is the correct or incorrect output, how to identify an apple in various conditions, and how to measure its size as accurately as possible. Our team travelled to growers and collected thousands of apple images to teach the system what an apple really looks like and then created a thousand individual apple annotation labels for hand selected images. Our pre-trained model was then retrained, which is known as transfer learning.
We used a pre-trained model to detect individual apples and draw masks and bounding boxes around them. The geometric information of the masks and bounding boxes were translated into real-world dimensions, such as millimetres and inches.
We found that humans couldn’t always replicate a photo with the perfect angle and pitch using a mobile phone. The solution: we applied a perspective transformation to stretch the bin top into a known frame position, mimicking a top-down view.
A histogram was produced from the bounding box width data to present how many fruits of each size category were detected. The two most useful points of information for apple growers based off histogram data included mean size and standard deviation. The mean was the average apple size, while the standard deviation gave an indication as to how much variation in apple size existed throughout the top layer.
Spectre could now estimate the size for the entire bin by taking the detected fruit on the top layer of the processed full apple bin with accuracy.