2D Camera Integration

Different cameras can be integrated in the CPRog software. This allows to pick parts with a not precisely defined position.

Currently plugins are available for:

• IFM O2D, a compact camera with integrated image processing
• Matrox camera systems

Both systems aquire an image and process it internally. Then they send the target position via Ethernet to the CPRog software. Here they are received by an plugin and stored in a variable. This variable can be used in a motion.

German documentation is available here (german): Dokumentation_PlugIn_IFM-O2D_V04.pdf (DE) The english version of that same documentation is being worked on at the moment.

---

UNDER CONSTRUCTION. Currently the English translation of the documentation linked above is being generated

Scope

This document explains the use of an IFM 3D camera in CPRog.

Safety

• Caution! Personal safety has to be ensured during operation.
• This is especially relevant during configuration and set up of the camera application. All motion has to be carried out at slow speeds.
• The operator has to be ready to stop the robot
• It is recommended that all programs are tested in the simulation prior to moving the robot.

Mechanical and electrical setup

Preferred camera mounting position

The camera has to be mounted at ample height above the work-pieces. Care has to be taken to avoid any collisions. For acceptance cone and minimum distance from camera, please look at the IFM-Documentation.

• The camera should be mounted parallel to the coordinate axes of the robot, if at all possible. - That means either overhead, in front or from the side. This will simplify calibration significantly (see image on the right).

• Power requirements: Connect the leads of the black power cable to a 24V supply:
  • blue lead: GND
  • brown lead: 24V (please verify the pin configuration with the IFM-Dokumentation).

PC Network Configuration

PC network address configuration

• Plug ethernet cable into camera and PC.
• Set IP of PC to a free IP in the 192.168.0/24 range, e.g. 192.168.0.50
• Camera IP is factory configured for 192.168.0.49. So do not use this particular IP on your computer. It belongs to the camera already and you don't want them to be the same.

Camera configuration

IFM Camera TCP/IP setings

The image processing is done entirely in camera. To recognize a work-piece an "application" has to be set up containing the model of the work-piece. The camera settings are configured using the IFM Software efector dualis E2D200.

Communication with the CPRog-Plugin requires the following settings:

• General TCP/IP settings: The TCP/IP address and port configuration have to remain the IFM factory defaults.
• An example camera configuration can be downloaded here - it is set up to recognize a 2 Euro coin in our lighting conditions and camera set up. (Your mileage may vary. This file is provided to make sure that parameters, such as IP and result output are correct.) To get it to actually recognize a coin, read the IFM documentation. The file has to be unzipped before loading it by right clicking on an unused folder in the "Applications" dialogue and selecting "Download to Sensor" followed by right clicking on the same folder and selecting "Activate". Not that the IP configuration of the camera is also updated (to the defaultsettings) when doing that (192.168.0.49).

(Communication with TinyCtrl on an embedded Linux computer requires a different IP address due to the default configuration of that embedded Linux computer, which is itself configured for IP 192.168.4.11 by default. Hence the camera would have to be in the same IP range, e.g. 192.168.4.49. Initially, to simplify the setup, the camera configuration will always be done in Windows and at the very end the IP will be reconfigured to match the IP range of the network adapter in the Linux embedded computer.)

IFM Protocol Version settings

• The Protocol Version has to remain in the factory default setting.

IFM IO configuration

• Select you newly created project and click on the edit button on the LEFT.
• Then click on "Process Interface" and "Change Settings". The window "IO configuration" appears.
• In the "TCP/IP settings" set
  • "Result Output" to On
  • "Model Detail Output" to On

Plugin Installation

The plugin has to be installed in the Data\Plugins folder as described below for CPRog on Windows and also for the embedded computer (Linux). If the communication between your robot and the Windows PC is via the white PCAN-USB adapter, you just need to follow the CPRog instructions and can skip over the TinyCtrl instructions. If not, you have an embedded computer built into your robot or the control cabinet. The software running on that embedded computer is called TinyCtrl. In that case you need to follow both, the CPRog and the TinyCtrl instructions below.

CPRog

File:IFM camera plugin directory.png

IFM Camera TCP/IP setings

A plugin is required to use the IFM camera. It connects via ethernet to the camera and translates and transmits the data received from the camera to the robot control electronics.

Download, unzip and copy the following files in the zip file below to C:\CPRog\Data\Plugins\

• ConfigSmartCameraIFMO2D.zip
• (you will have received these files from CPR)

An example program (Example_IFM_Camera.xml) can be saved to C:\CPRog\Data\Programs\ and unzipped.

TinyCtrl

For TinyCtrl only the xml file contained in ConfigSmartCameraIFMO2D.zip is required.

It needs to be copied into the folder /home/root/TinyCtrl/Data/Plugins on the embedded computer.

After that the robot has to be restarted. Here is a guide that explains how to gain access to the embedded computer to copy files.

The IP address in the xml file in the link needs to be 192.168.4.49.

Use of the plugin within a robot program

File:CPRog Program Editor Camera PLugin Commands.png

Commands to get the camera up and running

Three commands have to be inserted in the Program editor to get the camera up and running. These commands can be added via the menu item "Advanced Commands"

• In the first line a position variable is defined. This variable is used to store the target position.
• in the second line a Number variable is defined. This variable is used to store the model type. Once the camera has recognized an object it is set to -1.
• The command "PlugInTargetPos" calls a plugin that returns a target position. It takes the following arguments
  • Name of the plugin to be used (here "PluginSmartCameraIFMO2D").
  • a variable for the target position
  • a variable for the model type.
• The plugin overwrites only the X, Y, Z and A (=rotation around Z) values. The rotation values B and C remain as they were. Therefore, upon definition of the variable the robot should already have been positioned so that it can grip the object with the current B and C values.

The example program uses the plugin. It moves the robot to a starting position and asks the plugin for a target position.

When the variable "modelclassvar" becomes -1 a small up and down movement is carried out.

Otherwise the robot will move to the target position by using the command "LinearByVariable".

Plugin Configuration Dialogue

In general, a new program should always be run in simulation prior to using it on the actual robot. To run the program in simulation simply click the plug button in CPRog to disconnect the robot. As soon as the Status light in CPRog changes (from red or green) to grey, you are in simulation mode.

When running the program for the first time, the plugin configuration window opens. Do not close this window. Keep it open! If an error message appears, confirm the error message with "OK", "yes" or "no" Here the camera position needs be configured.

• This is the place where the IP-Address of the camera, which has been left at the default setting earlier in this how-to (192.168.0.49), has to be entered.
• The default settings in this configuration window are identical to the IFM defaults.

• Pixel-to-coordinate-conversion: The camera outputs the object position results in pixels and hands it over to the plugin. To calculate the robot target position the pixel values have to be multiplied with the scaling factor. The scaling factor is dependent on the distance between camera and the surface the camera looks at (see next section).

• Geometry Setup: The position of the camera has to be defined relative to the robot. This is necessary to transfer the target position (in camera pixels) into an absolute position in the robot coordinate system.

• Origin: The position of the camera in the robot coordinate system (x/y/z value).

• Look Vec(tor): The viewing direction of the camera. In the image above, the camera looks down onto the work surface that the robot sits on, hence it looks in negative Z direction.
• Up Vec(tor): This vector defines the rotation of the camera around the target axis, i.e. top and bottom in a 2D image.
• Pick Distance: Is the distance between the top of the work-piece and the camera.

• Example: Work-piece located in the center below the camera would produce a target position (120, 120, 35)

• Test Configuration: For testing purposes the plugin can be configured so that it produces a target position, even when the camera is disconnected. This is so that a program can be tested in simulation mode: When the tick before "Override Camera" is set target position and model class defined in the line below is returned by the plugin.
• Status: Here the status of the camera connected is shown, i.e. the connection status and the current target position in pixels and mm.

• To apply any changes made in this window, the "Update" button needs to be clicked. The changes will then be uploaded to the camera and also saved in the camera configuration file, so that they are not lost after restart.

Status

During operation the values that the camera returns (position and model type) can be checked in the in the "variables" tab at the bottom of CPRog.

Calibration

Note: Please always set the "Pick Distance" initially, so that there are several cm of space between surface and robot arm. Set the override percentage to a low value, so that the robot moves slowly while testing. Be prepared to stop the program by clicking the "stop" button (or using the emergency stop).

The values in the image below are only examples.

The followings steps need to be carried out for a successful calibration

1. Mount the camera and write a motion program that uses the camera (or use the example program provided above).
2. Manually measure the camera position in the robot coordinate system using a ruler and enter the results in the fields "Origin", "Look Vec", "Up Vec" und "Pick Distance".
3. Center point: Test whether the robot grips a work-piece located in the center of the field of view of the camera. It is significantly easier to configure a camera that is mounted parallel or at right angles relative to the robot coordinate system. Odd angles complicate the calibration. Ideally the camera is located parallel to the work surface looking down on it and not looking down at the work surface at e.g. 60 degrees.
   • Place an object directly below the center of the camera, so that the pixel position 320 and 240 is shown in the status section of the plugin window. That is the center of the field of view of the camera.
   • Jog the robot to above that position. Axis 5 has to be orthogonal to the work surface (parallel to Z). Now adapt the values in the fiel "Origin", so that they reflect the robot position.
   • Leave the work-piece where it is and test a program that uses the camera (you could use the example program provided above). If necessary correct the "Origin" values.
4. Orientation: Check the orientation of the camera
   • Move the work-piece slightly in X direction. Check whether the position shown in the "status" section of the plugin window changes in the right direction. If it does not, the "Up Vec" needs to be updated to reflect the correct position of the camera.

Calibration

Note: Please always set the "Pick Distance" initially, so that there are several cm of space between surface and robot arm. Set the override percentage to a low value, so that the robot moves slowly while testing. Be prepared to stop the program by clicking the "stop" button (or using the emergency stop).

The followings steps need to be carried out for a successful calibration

1. Mount the camera and write a motion program that uses the camera (or use the example program provided above).
2. Manually measure the camera position in the robot coordinate system using a ruler and enter the results in the fields "Origin", "Look Vec", "Up Vec" und "Pick Distance".
3. Center point: Test whether the robot grips a work-piece located in the center of the field of view of the camera. It is significantly easier to configure a camera that is mounted parallel or at right angles relative to the robot coordinate system. Odd angles complicate the calibration. Ideally the camera is located parallel to the work surface looking down on it and not looking down at the work surface at e.g. 60 degrees.
   • Place an object directly below the center of the camera, so that the pixel position x=0 and y=0 is shown in the status section of the plugin window. That is the center of the field of view of the camera.
   • Jog the robot to above that position. Axis 5 has to be orthogonal to the work surface (parallel to Z). Now adapt the values in the fiel "Origin", so that they reflect the robot position.
   • Leave the work-piece where it is and test a program that uses the camera (you could use the example program provided above). If necessary correct the "Origin" values.
4. Orientation: Check the orientation of the camera
   • Move the work-piece slightly in X direction. Check whether the Position shown in the "status" section of the plugin window changes in the right direction. If it does not, the "Up Vec" needs to be updated.
5. If you do not have an embedded linux computer as part of your control electronics, you can now reset and enable the robot and try out the program in the real world. You may have to adapt the scaling factor, if you find that the robot does find the center point but does not move far enough in X or Y direction, when the recognized object is not in the center.
6. If you do have an embedded Linux computer, take a screenshot of the settings of the plugin configuration dialogue or write them down. You will need them, if you have an embedded Linux computer as part of your control electronics.

TinyCtrl

The embedded linux computer, which is part of the robolink DCi robot or can be purchased as an option for the DIN rail control electronics, uses a program called TinyCtrl. TinyCtrl is the Linux equivalent to CPRog. The embedded computer is used so that a robot can be run without an external Windows PC.

Configure new IP address

When an embedded computer is used, the camera needs to be connected to the embedded computer, after it has been configured as described above. The first step to connecting the camera to the embedded system is to configure the correct IP address (192.168.4.49):

1. Close CPRog
2. Start the IFM Object Recognition software and connect to your sensor, which should still be connected to your Windows PC (in this tutorial that is IP 192.168.0.49).
3. Go to "Applications" and select the tab called "Network Parameters" in the lower part of the window.
   • Configure IP address to 192.168.4.49
   • Configure Subnet Mask to 255.255.255.0
   • Configure Gateway to 192.168.4.201
   • Click "Assign". The camera will now loose connection to your windows PC.
   • Connect the green LAN cable between Camera and Embedded Linux Computer. The correct port to use is the lower (ETH1) port of the DIN Rail embedded computer or the left LAN port of the robolink DCi embedded computer.
   • (The IP configured here will later on be referenced in the plugin configuration file. The second network adapter (ETH1 of the Linux embedded system) is by default configured for 192.168.4.11. So do not use that particular IP here))

Install and update the configuration file

Now connect to your Linux embedded computer via LAN.

• Use any of the methods outlined in this link to edit the configuration file located in /home/root/TinyCtrl/Data/Plugins/ConfigSmartCameraIFMO2D.xml. The FTP method is usually easiest for someone with no prior experience in Linux terminal.
• Find this line in the configuration file:
  • <Cam0Geometry MaxWaitTime="5" ScaleX = "0.25" ScaleY="0.25" OriginX="120" OriginY="120" OriginZ="335" LookX="0" LookY="0" LookZ="-1" UpX="0" UpY="1" UpZ="0" ZDistance="300" SimX="350" SimY="-100" SimA="350" SimModelClass="4"/>"
  • Update the OrgingX, OriginY, OriginZ, LookX, LookY, LookZ, UpX, UpY, UpZ values and the ZDistance values with the values that you have noted down or made a screenshot of in the calibration step
    File:IFM camera config.png
    .

• After editing and saving that file (ConfigSmartCameraIFMO2D.xml) into the directory /home/root/TinyCtrl/Data/Plugins/ on the Linux computer.
• Power cycle the robot.
• Start CPRog and upload the unzipped example program that you have used earlier in this tutorial.
• Check, if the robot moves to the correct position when the coin is placed in the center below the camera.
• Check, if the robot moves too far or not far enough, when placing the coin outside the center in X and Y direction.
  • You will likely need to change the scaling factors (ScaleX and ScaleY) in the config file on the Linux computer (ConfigSmartCameraIFMO2D.xml) to make the robot move the right distance, when the work piece is located off-center. The scaling factor is essentially the factor for the conversion between pixels and mm.