Project

Rifle. The rifle used in our test is a softair replica of an an Автома́т Кала́шникова, tr. Avtomát Kaláshnikova, better known as AK-47. One may wonder why we used a replica of an automatic rifle instead of a sniper rifle. Simply because the University can't provide us any gun and we had to use one of our own.

Rifle housing. We have designed and built a MDF base structure to support the rifle which allows to turn it smoothly around the pitch and yaw axes. The purpose of the rifle housing is to keep the gun in a fixed position during the first phase of pose estimation, and then to allow the tracking of the rifle angular movements. At the bottom of the structure a box, containing the power supply and the hardware to link the processing unit, a Raspberry Pi, with all the input/output interfaces such as angular sensors, motors, laser projector and LCD display, has been placed. One of the main structural challange that led us to create the structure as it has been done was related to the angular sensors displacement, in particular, how to place the sensors in such a way that both the wiring of the sensors and the mobility of the rifle were easy without any hindrance, in order to make them work correctly, on the bottom and the left side of the structure were made two sites to host the sensors.

Printed circuit boards. After the first phase of testing where the single tasks like sensors communication, power supply, LCD communication, motor driving, etc.. were implemented and tested with simple hardware without paying attention to the cleaning and ordering of the prototype, we have decided to make it better by building an ad-hoc designed PCB used as a main "interfacing" board to link all the mentioned input/output interfaces together. Furthermore, we decided to make other two PCBs for the sensors in order to allow fast replacement and maintenance everytime it would have been necessary.

Motors and laser structure. For the vision based estimation of the relative rifle position with respect to the cameras framing the target, in particular, for implementing an active stereo setup (see below for a detailed explanation) a structure mounted on the rifle able to move a laser projector over the target by known steps was essentially needed. To achive this we have decided to build a structure, which allows to move the laser projector with two stepper motors. Due to the high distance between the rifle and the target a very small and precise step is needed to achive as much accuracy as possible for the laser pointed point. One big problem is the backlash, associated with the multi-stage planetary gearbox, which is required to be estimated and compensated by moving the structure in such a way to limit its impact. Considering the available budget and the specifications of the project, our choice was a couple of Crouzet-motors steppers equipped with an external gearbox and the Allegro A4988 driver to control them. Another important challange was to avoid any translation of the laser during the steps, in others words a fixed point of rotation around the pitch and yaw axes was needed. To achive this we built the structure in figure with a 3D printer. In particular, in the pink and grey "bowl" the two steppers have been placed and the wings were used to stabilize them through clamps. The blue/transparent cylinders rapresent the rotation axes of the two motors and they intersect in the center of the yellow structure, which is the centre of rotation and thus the place where the laser has been placed. Furthermore, a micro-alignment and adjustment system (2 simple screws coupled with 2 springs), was provided for micrometric adjustment.

The assisted aiming system works by estimating the pose, both rotation and translation, between the rifle and the target. As briefly mentioned above our proposal is to employ a vision-based pose estimation building an active stereo setup which involves one laser projector and a stereo camera located in the close proximity of the target (this one can be tought attached to an UAV drone, driven in front of the target). With this setup one can, in principle, obtain an accuracy in the pose estimation much better than any other absolute pose measurement (i.e. with GPS). The only one constraint is the ability for the laser beam to reach the target foreground without spreading too much, to achieve this, a collimated laser projector that can ensure the wanted performance and can produce a thin spot even at a couple of hundreds meter of distance has been used.

Target recognition. The aim of a stereo camera setup is to recognize the target, that in our case it is supposed to be planar, and to provide a 3D reconstruction obtained through triangulation, the scale ambiguity is usually solved with the perfect knowledge of the translation between the two camera center provided by the streo camera manufacturer. After the scene reconstruction, a robust Least-Square plane fitting technique provides the equation of the plane on which the target lies. Due to lack of resources, we actually did not employ a stereo camera (like Intel RealSense D400) but we used two different cameras in an RGB-D stereo setup, in particular, a Microsoft Kinect v2.0 and a Raspberry Pi Camera Module V2, the reason why we use a RGB-D stereo setup is to solve the scale ambiguity estimating the depth information instead of knowing the translation between the two camera center, a detailed explanation is given below.

Calibration. In order to use the cameras to make measurements in the 3D space, and so to realize a photogrammetry setup, they have to be calibrated and we have done that for different resolutions and aspect ratios. Our calibration method (based on Zhang's calibration) also takes in account and compensate the lens distortion of actual cameras that introduces a deviation from the pinhole model of a camera. The calibration process has to be made una-tantum (factory calibration) and gives as result the intrinsics of the camera. Relative pose between two cameras is then robustly estimated by the usual frame-based methods based on points/features-correspondences.

Scale recovery. As mentioned above the requirement of having a stereo camera without the knowledge of the translation between the camera centers is not sufficient in order to reconstruct the relative pose between cameras. Infact, due to the well known depth/speed scale ambiguity, the translation existing between the two cameras can be estimated up to a scale and thus it is simply extracted as a versor. In other words the estimation of the rigid body transformation between the two camera frames that has six degrees of freedom (both the rotation and the translation have three degrees of freedom) can be done only up to five degrees of freedom, hence we lose one degree of freedom which is the scale (the norm/length of the translation vector) as shown in the image on the right. Different setups provide the same estimation since we do not know the length of the translation vector (t1 and t2 in red on the image). By introducing a depth camera, we can solve this ambiguity estimating the depth and hence fixing a constraint on the norm/length of the translation vector. To do that we have proposed an innovative, straightforward algorithm that perform the least square estimation of the magnitude of the translation using the additional depth information.

Active stereo based estimation. In order to estimate the pose between the rifle and the target we have proposed to build an active stereo setup, where a projector is used to act like an "inverse-camera" since the geometry of a camera and a projector is essentially the same. This type of projectors taking the name of "laser scanners" are usually employed in photogrammetry applications as LIDARs. Professional laser scanners are realized using a fixed laser beam source, typically a LASER photodiode, passing through a system of at least two mirrors precisely actuated by galvanometers (voice coils, as in hard disk drives heads actuator). Due to the cost of professional laser scanners, our approach was to realize the laser scanner by directly moving the laser beam with two strongly reduced stepper motors. This, low cost, choice obviously has a direct impact on the accuracy that can be obtained with such a system. To use the laser projector as an "inverse-camera" for the pose estimation we have built a mathematical model in order to obtain a description of the projector coherent with the pinhole model adopted for the cameras and hence such that we can use either cameras or projectors independently as a black box for the pose estimation algorithm.

Rifle-target pose estimation. In order to improve the noise immunity and to exploit some useful constraint about the geometry that is created between the three cameras (aka two cameras and one projector), the relative pose between rifle and target is estimated two times - namely it is computed bwtween projector and one head of the stereo camera, and between projector and the other head - simultaneously. The recovery of the pose works this way:

1. The sniper must align - in a very rough manner - the rifle towards the target
2. The laser is turned on and the projector is moved along a well-defined trajectory
3. This movement translates in a certain number of laser spots projected in the scene near the target
4. Two cameras must recognize the laser spots with an appropriate detection algorithm
5. Laser is turned off and it will never be used in the aiming-assisted process
6. The two poses are recovered exploiting the particular epipolar geometry, by points correspondeces

Once the two pose are estimated, and since we have also estimated a third pose - namely the relative pose between the two stereo camera heads - we have to ensure the overall consistency by the "alignment" of the three estimated relative poses. In fact this is not guaranteed, since every estimation is corrupted by noise, pixelization and various types of offsets. With the three relative poses estimates we have build up a pose synchronization problem, that is an optimization problem appositely realized for rigid-transformations. In other words, the relative poses must satisfy a composition constraint, that the estimated ones don't. Synchronization allows us to determine the nearest relative poses (wrt the estimated one) undergoing this constraint.

Bundle adjustment. Bundle-adjustment, namely the joint points/poses nonlinar refinement, ends up the pose estimation process. After BA, implemented using sba, the pose between the rifle and the target is known with the best estimate. This concludes the vision based pose estimation process. The recovered pose is used jointly with the information provided by the angular encoders (introduced below) to draw the virtual viewfinder on the display.

Sensing requirements. In order to move the virtual viewfinder on the display, following the movements of the rifle rod, some sensors must be used to track the pitch and yaw angular displacements. As mentioned at the beginning these angles should be measured with the highest resolution and accuracy as possible in order to fullfill the requirements in terms of distance between rifle and target. Typical angular sensors are optical encoders, but the resolution required to meet our specifications would make them extremely expensive and bulky. Thus, we have decided to employ a small, cheap and capable to meet our requirements (only for the prototype fixed distance) magnetic angular encoder.

Sensors. The chosen sensor for the rifle orentation is the Allegro A1335 which is a 360° contactless high-resolution programmable magnetic angle position sensor IC. It is designed for digital systems and is capable of communicating via different interfaces as either I2C, SPI, or SENT. The A1335 incorporates a Hall sensor IC that measures the direction of the magnetic field vector through 360° in the x-y plane (parallel to the branded face of the device). The A1335 computes the angle based on the actual physical reading, as well as any internal parameters that have been set, for example: the output dynamic range, output scaling, and filtering. The choice of this sensor has been based on the tradeoff between the cost of the sensor itself and the usability intended as angle reading performance in terms of accuracy, resolution, velocity, etc.. and the possibility to develop a simple firmware (for a detailed explanation of the sensor performance, architecture and working principle we refer the readers to the official datasheet).

Angle measurements and rifle kinematics. As briefly mentioned above, the main task of the sensing part is to provide an accurate angle measurement in both pitch and yaw referrend to the initial angle, at the system startup, set as zero. This angle measurement define a vector (in spherical coordinates) on the sensor reference that, once it is transformed in the gun barrel reference, define a ray (in cartesian coordinates) lying on the gun barrel itself that is used to define where the rifle is pointing .

Setup. As already said, all the input/output interfaces from the rifle side as the angular sensors, the stepper motors and the LCD have been connected and communicates throught a Raspberry Pi. On the target side, to communicate with the Microsoft Kinect we use a Windows 10 PC due to the compatibility constraint given by Microsoft. To resume, the overall processing system is composed by two Raspberry Pi and one PC. To make them interact we used a smartphone as WIFI hotspot. This solution was possibile because the distance between the rifle and the camera wasn't so big (around 30 meters in our experiments); for a bigger distance another solution is obviously required, for example using an ad-hoc WIFI network supported by hardware for long distance communication. Moreover, we wrote a communication protocol based on sockets in such a way that all the communication starts automatically when the RPis and PC turn on. The overall setup can be seen in the main image above.

Clock synchronization. To conclude, an important technical detail, especially, during the vision estimation process and the movement of the laser was to have a synchronized clock on all the three diveces (the two RPis and the PC). This was necessary because we wanted to couple the images shooted by the kinect and the RPi with the movement of the laser using the timestamp. To do that we had to establish a client-server architecture with a local NTP service for the synchronization because the service offered via internet has seconds of delay.

Softair replica of the actual rifle is compliant with italian regulation act, 18th April 1975 n. 110 which states that ASGs (air soft guns) must provide to the bullet at most 1J of energy. Since the bullets used have mass of 0.25g, then the magnitude of the velocity at the output of the rifle's barrel is v = 89m/s. Test site has been located 45°53′29″N 12°20′20″E, 46m a.s.l., where the acceleration of gravity is locally estimated as g = 9.8167 m/s^2. Over a distance of 30m between the sniper and the rifle, the Δy introduced by parabolic motion is about 55.8cm. Our implementation takes care and correct for the parabolic motion effect. Since the distance over which the bullet travels is relatively small, our implementation actually does not take in accout for any fluid dynamic effects, such as the Magnus effect.

Rifle housing. The structure to accomodate the rifle was designed using SketchUp. In order to do this, the rifle was precisely measured (since the replica actually does not comply with the original rifle design at all). The measurements were taken on a plywood sheet and then a proper mold was realized to support the subsequent phases of design. Each element of the structure was then exported in a 2D CAD format, imported in a CAM and realized with a 6 dof CNC machine over a medium-density fibreboard, coated with paper. Machinery was a fundamental process in order to meet the requirements in terms of accuracy in angular measurements. Sensors housing, infact, has been placed and realized precisely where we wanted to, avoiding any out-of-centre related issues. As a final touch, all the components were then painted in gunmetal gray fashion (RAL 7024), by using spray can color. The 2 DoF of the structure (pitch and yaw) are provided by a two arms rotating about a nylon pivot and a big circular bearing fixed to the base plate, respectively. The nylon pivot was designed and machined with a lathe.

Printed circuit board. Printed circuit boards were designed using Altium Designer 18.0. Before the design, we have realized a prototype on a perfboard using the wire-wrapping technique. We realized the first PCB ourselves, using a DIY photolitography followed by chemical etching technique. PCB was dual layer in through-hole technology, and we have to realize by hand all the vias and the holes using a drill press. We then have made some changes on the design and the new board was realized by PCBWay.

Printed 3D structure. The printed 3D structure was designed using Tinkercad, a free online modelling software for beginners very easy to use, it's like to use LEGO bricks. We point out that this solution it's not suitable for very complex projects, but it was the best choice to make a prototype without strong knowledge in 3D design and printing. An important aspect was about the resistance and stiffness of the material used to print the structure that has to support the weight of the motors without breaking or bending, thus avoiding shift of the rotation center. The material that best fit the requirements is the nylon but due to the high cost we have decided to build the structure using another material. The choices were between either the PLA, that is cheap and very resistent but it is very sensitive to heat and degradation, or the ABS which is cheap, very strong but flexible. Due to the heat produced by the motors and the laser we have decided to emply an ABS structure. Aware about the flexibility of the material, we have chosen to fill the stamp for the 50% and so to avoid bending and to reduce the cost at the same time.

Power supply and PCB housing. In our tests we supplied the system by the mains and the power supply was chosen to provide two different otuput voltages to supply respectively the motors H-bridges and the control electronics. To guarantee the functioning in the worst scenarios, the power supply has to provide 60W and was chosen to be a flyback isolated power supply. Power supply and PCB were then embedded into a 44 CE Range watertight junction box.

Measurements. In order to mathematically define the kinematics of the gun barrel, a certain number of rigid transformations had to be computed. Most of them contains distances and rotations between some keypoints on the rifle; such quantities have to be measured with the least uncertainty in order to not compromise the aim. Since it is very difficult to make such measurements with conventional instruments on the uneven surfaces of the rifle, we adopted a vision based approach. An high resolution conventional reflex camera with proper optic was employed to realize a set of quasi-ortographic photos of some regions of interest of the rifle. In this case, it is very important to take in account and correct the lens distortion and to play down the perspective effects in order to obtain accurate measurements. Also, a reference distance has to be known with the highest accuracy in order to obtain metric distances. The software used was ImageJ.

Kinect interface. Interfacing a Windows PC with the Kinect is made possibile by Kinect for Windows SDK 2.0 (although discontinued and very buggy). First tests were made using MATLAB with the Image Acquisition Toolbox to obtain useful results regarding the performance of the depth sensors and to evaluate the accuracy of the depth measurements. We realized a script that allows the user to pick a point in the RGB frame obtaining the corresponding depth. This is not trivial at all, since RGB and depth cameras centers do not match and taking in account for this displacement is mandatory. An introductory document about this can be the following. Later, a C# program was written using the SDK APIs to implement this fundamental coordinate mapping. This mapping is hard coded into every Kinect Sensor during the factory calibration and is accessible only by using the SDK provided by Microsoft. Coordinate mapping is a LUT whose dimensions are the one of the RGB frame (full HD) and the content of every entry is the corresponding pixel on the depth frame. An ad-hoc C# program was later written to acquire single RGB frames.

Aim assistance: Hardware.

• Rifle-side:
• Raspberry Pi 3 B, equipped with UNIX-like Raspbian 9 Stretch (2018-03-13) distro
• 7in 800x480 touch display
• Target-side:
• Raspberry Pi 3 B, equipped with UNIX-like Raspbian 9 Stretch (2018-03-13) distro and interfaced with a Raspberry Pi Camera Module v2
• Microsoft Windows 10.1809 x64 PC interfaced with Microsoft Kinect v2. USB 3.0 interface availability is mandatory.

Aim assistance: Software. Most of the functions implementing the assisted aiming system are written in Python 3.0 with the used of the OpenCV Library. Both rifle-side and target-side Raspberry Pis are equipped with Python 3.6.2, and OpenCV 3.4.1. Since we have used intensively SIFT that is free for non-academic use only with the licence from the creator, and therefore this module is no longer included in OpenCV, it must be added to the standard library including the package opencv_contrib. The windows 10 PC is equipped with both Python 3 and OpenCV provided by Anaconda. The IDE we have used is Spyder 3.3.3. The NTP server is provided by the Windows 10 PC, using Meinberg NTP server. Kinect interface is provided by Kinect for Windows SDK 2.0 (discontinued). The start at the boot of the main script has been implemented using the "autostart system".

Sensors firmware. To use the fully capabilities of the sensors and to interface them with the Raspberry Pi we develop a simple firmware in C based on an hand made C library and the well known WiringPi library, the developed library allow us to acces to the internal programmable microprocessor-based signal processing features, including temperature compensation and gain/offset trim, as well as advanced output linearization algorithms, providing an extremely accurate and linear output for both end-of-shaft applications as well as off-axis applications.

EMI. The sensors can be used with different ways of interfacing as: I2C, SPI, SENT, Machester... We have decided to employ a SPI communication which is a single-ended and used for short-distance. The setup is in full duplex mode with a master-slave architecture with a single master. Although SPI requires more cables than I2C we have based our choice manily on robustness against disturbance. Since data transmission in SPI is done on sampled data on the clock's rising/falling edge, instead of a level sensitive data on the clock's active (high) level, this provides an improved robustness against disturbances over relatively long distances (about 30 cm), in hostile environments. The cable, infact, runs next to the motor phases, where a highly impulsive current flows. Moreover, the communication's speed was slowed down to 1 Mbps and we have intensively tested the link QoS in harsh electromagnetic environments and we never encountered communication fails.

Presentation site. To conclude we made a fully responsive presentation site through HTML5-CSS3_JavaScript based on a fully customizable free-to-use HTML5UP template.

As widely discussed and proved, our prototype is working in an enviroment with reduced distance and a fixed stereo RGB-D setup in the proximity of the target. To go beyond this limitations a better, specific and obviously expensive hardware is required. In this section we try to give some tips to develop and improve the Sniper Remote Scope. The first, simple, improvement to the prototype mobility is to supply the whole system through a battery pack instead of using the mains. Secondly, to increase the speed of the whole computation and to allow easy software maintainability and upgrading a good thing should be to implement the whole software in ROS that is becoming the de facto standard in robotics. Furthermore, instead of having a fixed stereo RGB-D setup, it would a good idea to use an UAV drone (equipped with the camera setup), controlled by the sniper. Obviously, increasing the distance it is necessary to use more accurate angular sensors on the rifle side and, maybe, to find a different way to move the laser for the vision estimation process.