3.3 – How Does Zumi See

Self-driving cars need more than obstacle detection sensors. They require an understanding of the surrounding environment to navigate. Humans rely on eyes and ears to detect potential dangers ahead, such as pedestrians, cyclists, and other vehicles. Self-driving cars also need this capability.

This module teaches you how to access Zumi’s camera, take pictures, and display videos. You will learn to import camera and vision libraries containing the code to take, modify, and display images.

Step 1: How Does Zumi See?

To navigate our world, self-driving cars need more than just obstacle detection sensors. They need to detect and recognize the different elements in the environment, such as pedestrians, vehicles, and other objects.

Step 2: Take a selfie

The first step is to use Zumi’s camera to take a picture and display it on the screen. You will need to import the camera and vision libraries before running the code.

from zumi.zumi import Zumi
from zumi.util.screen import Screen
import cv2
import time
from zumi.util.vision import Vision
from zumi.util.camera import Camera

zumi = Zumi()
camera = Camera()
screen = Screen()
vision = Vision()

Step 3: Cheese!

Similar to taking an actual picture, you will need to count down to prepare yourself. Once you are ready, run the code, smile, and see yourself on the Zumi screen!

camera.start_camera()
print("3...")
screen.draw_text_center("3...")
time.sleep(1)
print("2...")
screen.draw_text_center("2...")
time.sleep(1)
print("1...")
screen.draw_text_center("1...")
time.sleep(1)
screen.draw_text_center("Cheese!")
image = camera.capture()
camera.close()
screen.show_image(image)

Step 4: Displaying images in Jupyter

Alternatively, you can display the picture on Jupyter Notebook. The picture will appear in color.

camera.start_camera()
print("3...")
screen.draw_text_center("3...")
time.sleep(1)
print("2...")
screen.draw_text_center("2...")
time.sleep(1)
print("1...")
screen.draw_text_center("1...")
time.sleep(1)
screen.draw_text_center("Cheese!")
frame = camera.capture()
camera.close()
camera.show_image(frame)

Step 5: Camera functions

Before taking a picture, you must turn on the camera with start_camera(). You cannot take an image without the camera stream. The red light indicates that the camera is on. Use capture() to take a picture, save it in a variable, and display it later.

frame = camera.capture()

Remember to turn off the camera with close(). If you do not run close() and try to run start_camera() again, you will get an error. If this occurs, save and close the notebook to force the camera to turn off.

Step 6: Show the image

Zumi processes images based on the pixel values of color, not the image’s visual representation. Write code to capture an image and save it to a variable. Then, print the variable to see what the image looks like.

camera.start_camera()
image = camera.capture()
camera.close()
print(image)

Humans, on the other hand, require visual representations of images. To display the image in Jupyter, use the show_image() function.

camera.show_image(image)

Step 7: Changing Colorspaces

Zumi can convert images to different colorspaces to aid in processing. This step demonstrates how to convert an image to grayscale, HSV, and inverted colorspaces. Grayscale images are made up of only gray pixels, making them faster to process than colored images. HSV images are useful for detecting or tracking certain colors, as they provide information about color intensity and shadows. Inverted colorspaces are just for fun and invert the tones of the colors in the image.

Grayscale

• Grayscale images are composed of gray pixels, making them faster to process since there are no other colors.
• To convert an image to grayscale, use the following code:
• Grayscale images can be used for scanning QR codes.

  camera.start_camera()
  img = camera.capture()
  camera.close()
  gray = vision.convert_to_gray(img) # Convert it to gray
  camera.show_image(gray)

HSV

• HSV stands for hue, saturation, and value.
• This color space is useful for detecting or tracking certain colors since each pixel can convey information about the color’s intensity and the presence of shadows.
• To convert an image to HSV, use the following code:

  camera.start_camera()
  img = camera.capture()
  camera.close()
  hsv = vision.convert_to_hsv(img) # Convert it to HSV, hue saturation and value
  camera.show_image(hsv)

Inverted

• This filter inverts the tones of the colors in an image. For example, lighter areas become darker, and darker areas become lighter.
• To invert the colors in an image, use the following code:

  camera.start_camera()
  frame = camera.capture()
  camera.close()
  invert = cv2.bitwise_not(frame) # invert the colors
  camera.show_image(invert)

Step 8: Resolution

You may have noticed that the images displayed on Zumi’s screen appear less detailed. This is because the OLED screen has a resolution of only 128 pixels in width and 64 pixels in height! In case you’re unfamiliar with pixels, think of them as tiny squares of color that make up an image. The first image we showed you has a resolution of 770 pixels wide and 600 pixels tall. These pixels are so small that they’re almost invisible unless you zoom in.

Now take a closer look at the eyes in the second image. You can clearly see the individual pixels that make up the image. There are 770 pixels in each row and 600 in each column! An image with even more pixels would be considered high-resolution. However, the OLED screen’s resolution is relatively low.

Step 9: Changing Resolution

Although the resolution of Zumi’s camera cannot be changed, the resolution of captured images can be increased. The code in this step allows for the user to specify the width and height of the captured image, which will stretch, shrink, or enlarge the image.

width = 160 # <-- CHANGE ME!
height = 128 # <-- CHANGE ME!
camera = Camera(width,height) # Let the camera know what changes you are making!
camera.start_camera()
img = camera.capture()
camera.close()
camera.show_image(img)

Step 10: Challenge – Video

Videos are a series of images displayed in quick succession. This step shows how to capture and display a video using a for loop. To avoid crashes, the code includes a try-finally statement, which will automatically close the camera if an error occurs.

camera = Camera()
camera.start_camera()
try:
for x in range(30):
# TODO Take a picture
# TODO show the picture
camera.clear_output() # Clear the output for the next image to show
finally:
camera.close()

Step 11: Review
In this module we have learned how to use Zumi’s camera and how it is a powerful tool for understanding the environment. Various image processing techniques can be used with Zumi.

Demo Video

Questions