Dennis Groß

The digital transformation of the workplace has only just begun. The notion that you have to move to Silicon valley to get employed by one of the world-class organizations is just not the case anymore.

Platforms like Fiverr and Upwork give freelancers the possibility to advertise their services to millions of customers remotely. That offers an excellent opportunity for people who want to travel around the world and still want to earn money.

Remote freelancing allows people from third-world countries to easily participate in the western world markets without leaving their homes. Remote freelancing from a third-world country allows freelancers to improve their lifestyle. This will also lead to economic growth in these third-world countries, especially in areas with high unemployment rates.

While businesses compete for local talents, remote freelancers give smaller startups a larger talent pool to choose from. Instead of hiring a graphic designer in the west, startups gain access to a far broader and deeper talent pool with these freelancer platforms than those who limit themselves to one geographic area. And for managers, organizing and coordinating a remote team’s work is crucial to winning recognition and advancement in the coming years.

So what will change in the future? I think that the prices for services that can be done remotely will drop, and they will be more and more outsourced in third-world countries. People who can do their work remotely may move to nicer places and do not need to live where their employer is located. This could have a quite interesting effect in, for example, Europe. Nowadays, low-wage countries move to North-European countries to earn more money while North-European citizens move to South-European to enjoy a friendlier climate.

Over the past few years, medical diagnostic apps are on the rise. Rapidly emerging technologies used to diagnose diseases result in more personalized patient care. About 70% of medical decisions are supported by diagnostics [2]. However, short of specialists and relatively low diagnostic accuracy calls for a new way of diagnostic strategy, in which deep learning may play a significant role. Smartphones and wearable devices can play a key role in health monitoring and diagnostics. They already support remote diagnostics and decrease the workload of GPs.

Diagnostic Apps

Modern smartphones are fitted with several sensors. These sensors allow for the sensing of several health parameters and health conditions.

Respiratory sounds are important indicators of respiratory health and respiratory disorders. When a person breathes, the sound emitted is directly related to air movement, changes within lung tissue, and the position of secretions within the lung. Depending on the sound, it is possible to monitor asthma, to record respiratory sounds for snoring and sleep apnea severity, detect respiratory symptoms like sneeze and cough, to detect bronchitis, bronchiolitis, and pertussis, to record wheezes in pediatric populations, and to detect the chronic obstructive pulmonary disease (COPD).

Leukocoria is an abnormal white reflection from ophthalmologists’ retina to detect several different eye diseases. As well as being an early indication of retinoblastoma, a pediatric eye cancer, leukocoria can also be a sign of pediatric cataract, Coats’ disease, amblyopia, strabismus, and other childhood eye disorders [4]. A portable 3d-printed device connected to a smartphone makes precise images of the retina to detect back-of-the-eye (fundus) disease at a far lower cost than conventional methods [3]. It is also possible to analyze images of the eye taken with this retinal camera to detect diabetic retinopathy, one of the leading causes of blindness.

Heart rate variability (HRV) is a measure of variations in the time intervals between your heartbeats and describes how “uneven” your heart beats. This metric can be used for different purposes:

• to measure stress levels
• to diagnose chronic health problems
• to assess the immune system
• to predict the recovery time after a severe illness

Ear and mastoid disease can easily be treated by early detection and appropriate medical care.

Skin diseases are widespread nowadays and spreading widely among people. The resolution of smartphone cameras has such a high resolution nowadays that these kinds of diseases can be detected [5].

In the modern world, psychological health issues like anxiety and depression have become very common among the masses.  Smartphone technology can be used to diagnose depression and anxiety [2].

And this is not the end of the medical usage of smartphones. Future sensors for analyzing sweat, isoline levels, and more are coming and will support more diagnostic applications for smartphones and wearables.

Datasets

The lack of large training data sets is often mentioned as an obstacle. However, this is only partially correct. Nowadays, hospitals are huge data storages, and databases in, e.g., radiology, are filled with millions of images. There are also large public data sets available on the internet.

What ML model for what task?

A survey name ‘A Survey on Deep Learning in Medical Image Analysis’ concluded that the exact deep learning architecture is not the most important determinant in getting a good solution. More important is expert knowledge about the task that can provide advantages beyond adding more layers to a CNN. Novel data preprocessing strategies also contribute to more accurate and more robust neural networks [6].

Problems

There remain still some problems with the accuracy of such systems. 98% accuracy of a diagnostic system can be quite expressing, but when we scale this up to a user base of millions, we could overrun our health systems with wrong diagnosed patients. So it is essential to make these diagnostic systems more and more robust.

References

[1] Majumder, Sumit, and M. Jamal Deen. “Smartphone sensors for health monitoring and diagnosis.” Sensors 19.9 (2019): 2164.

[2] https://www.bupa.com/newsroom/our-views/the-future-of-diagnostics

[3] https://medicalxpress.com/news/2019-06-portable-device-eye-disease-remotely.html

[4] Munson, Micheal C., et al. “Autonomous early detection of eye disease in childhood photographs.” Science advances 5.10 (2019): eaax6363.

[5] Chan, Stephanie, et al. “Machine Learning in Dermatology: Current Applications, Opportunities, and Limitations.” Dermatology and therapy (2020): 1-22.

[6] Litjens, Geert, et al. “A survey on deep learning in medical image analysis.” Medical image analysis 42 (2017): 60-88.

Governments around the world are increasingly investing in autonomous military systems. Many are already developing programs and technologies that they hope will give them an edge over their adversaries.

AI in Warfare

Target Systems Analysis (TSA) and Target Audience Analysis (TAA) are intelligence-related methods to develop a deep understanding of potential areas for operations. Detection of military assets on the ground can be performed by applying deep learning-based object detectors on drone surveillance footage. For military forces on the ground, the challenge is to conceal their presence as much as possible from discovery from the air. A common way of hiding military assets from sight is camouflage, for example, by using camouflage nets.

Algorithmic Targeting: Today autonomous weapon platforms are using computer vision to identify, track targets, and shoot targets. These algorithmic targeting technologies are nowadays so precise, that even the military adds randomness to the targeting process to spread the bullets over the target. Otherwise, the bullet would just fly through the already generated holes.

Problem

These previously mentioned AI systems outperform human experts dramatically. But there are still major issues with these technologies. These systems are vulnerable to adversarial noise. You can imagine this (adversarial) noise as blurry pixels on your Instagram photos, usually, these pixel perturbations are so small that none of your human followers will see them but when Instagram uses an image classification software to check your account for illegal posts, it could happen (in the WORST CASE) that Instagram will block you.

Manipulating these AI systems

So what can you do when you forgot your camouflage net or when you don’t want a huge hole in your battleship? Well, you could try to generate this previously mentioned adversarial noise and stick it on your battleship or airplane. So from the point of mobility, it would be much easier to print a specific pattern on top of your battleship to hide from adversarial drones as carrying camouflage nets.

Summary

In this post, I wanted to describe the current weaknesses of AI systems, especially in warfare. The biggest problem of the current AI technology is (not only in warfare), that they are quite vulnerable against (adversarial) noise. This should be considered before releasing fully autonomous war machines that could make huge damage because of perturbated pixels in input data.

Our totally wasted grid world agent calls us and asks if we can safely navigate him home. We use the WhatsApp location sharing feature to determine the agent location. Our goal is to message the agent a policy plan which he can use to find his way back home. Like every drunk guy, the agent does noisy movements: he follows our instructions only in 80% of the time. In 10% of the time, he stumbles WEST instead of NORTH and in the other 10%, he stumbles EAST instead of NORTH. Similar behavior occurs for the other instructions. If he stumbles against a wall, he enters the current grid cell again. He gets more tired of every further step he takes. If the police catch him, he has to pay a fee and is afterward much more tired. On the way back home it could happen, that he gets some troubles with the “hombre malo”, and in his condition, he is not prepared for such situations. He can only rest if he comes home or meets the “hombre malo”. We want to navigate him as optimal as possible.

Markov Decision Process

Because of the noisy movement, we call our problem a non-deterministic search problem. We have to do 3 things to guide our agent safely home: First, we need a simulation of the agent and the grid world. Then we apply a Markov Decision Process (MDP) on this simulation to create an optimal policy plan for our agent which we will finally send via WhatsApp to him. An MDP is defined by a set of states \(s \in S\). This set \(S\) contains every possible state of the grid world. Our simulated agent can choose an action \(a\) from a set of actions \(A\) which changes the state \(s\) of the grid world. In our scenario he can choose the actions \( A = \{NORTH, EAST, SOUTH, WEST\}\). A transition function \(T(s,a,s’)\) yields the probability that \(a\) from \(s\) leads to \(s’\). A reward function \(R(s,a,s’)\) rewards every action \(a\) taken from \(s\) to \(s’\). In our case, we use a negative reward of \(-0.1\) which is also referred to \(\textit{living penalty}\) (every step hurts). We name the initial state of the agent \(s_{init}\) and every state which leads to an end of the simulation terminal state. There are 2 terminal states in our grid world. One with a negative reward of \(-1\) in \((1,2)\) and one with a positive reward of \(1\) in \((3,3)\).

Solving Markov Decision Processes

Our goal is to guide our agent from the initial state \(s_{init}\) to the terminal state \(s_{home}\). To make sure that our agent does not arrive too tired at home, we should always try to guide the agent to the nearest grid cell with the highest expected reward of \(V^{*}(s)\). The expected reward tells us how tired the agent will be in the terminal state. The optimal expected reward is marked with a *. To calculate the optimal expected reward \(V_{k+1}^*(s)\) for every state iteratively, we use the Bellman equation:

\(V^*_{k+1}(s) = \max_{a} Q^{*}(s,a)\)

\(Q^{*}(s,a) =\sum_{s’}^{}T(s,a,s’)[R(s,a,s’) + \gamma V^{*}_{k}(s’)]\\\) \(V_{k+1}^*(s) = \max_{a} \sum_{s’}^{}T(s,a,s’)[R(s,a,s’) + \gamma V_{k}^{*}(s’)]\\\)

\(Q^{*}(s, a)\) is the expected reward starting out having taken action \(a\) from state \(s\) and thereafter acting optimally. To calculate the expected rewards \(V^{*}_{k+1}(s)\), we set for every state \(s\) the optimal expected reward \(V_{0}^{*}(s) = 0\) and use the equation.

As soon as our calculations convergent for every state, we can extract an optimal policy \(\pi^{*}(s)\) out of it. An optimal policy \(\pi^(s)\) tells us the optimal action for a given state \(s\). To calculate the optimal policy \(\pi^{*}(s)\) for a given state \(s\), we use:

\(\pi^{*}(s) \leftarrow arg\max_{a} \sum_{s’}T(s,a,s’)[R(s,a,s’) + \gamma V_k^{*}(s’)]\)

After we calculated the optimal policy \(\pi^{*}(s)\) we draw a map of the grid world and put in every cell an arrow which shows the direction of the optimal way. Afterward, we send this plan to our agent.

This whole procedure is called offline planning because we know all details of the MDP. But what if we don’t know the exact probability distribution of our agent stumbling behavior and how tired he gets every step? In this case we don’t know the reward function \(R(s,a,s’)\) and the transition function \(T(s,a,s’)\)? We call this situation online planning. In this case, we have to learn the missing details by making a lot of test runs with our simulated agent before we can tell the agent an approximately optimal policy \(\pi^*\). This is called reinforcement learning.

Reinforcement Learning

One day later our drunk agent calls us again and he is actually in the same state ;) as the last time. This time he is so wasted that he even can not calculate the reward function \(R(s,a,s’)\) and the transition function \(T(s,a,s’)\). Because of the unknown probabilities and reward function, the old policy is outdated. So the only way to get an approximately optimal policy \(\pi^{*}\) is to do online planning which is also referred to reinforcement learning. So this time we have to run multiple episodes in our simulation to estimate the reward function \(R(s,a,s’)\) and the transition function \(T(s,a,s’)\). Each episode is a run from the initial state of the agent to one of the terminate states. Our old approach of calculating \(\phi^{*}\) would take too long and it doesn’t convergent smoothly in the case of online training. A better approach is to do Q-Learning:

\(Q_{k+1}(s, a) \leftarrow \sum_{s’} T(s,a,s’) [R(s,a,s’) + \gamma \max_{a’} Q_k^{*}(s’, a’)]\\\)

Q-Learning is a sample-based Q-value iteration To calculate the unknown transition function \(T(s, a, s’)\) we empirically calculate the probability distribution of the transition function.

This time we choose our policy based on the q-values of each state and update the \(Q(s, a)\) on the fly. The problem with this policy is, that it exploits always the best \(Q(s, a)\) which is probably not the optimal policy. To find better policies, we introduce a randomization/exploration factor to our policy. This forces the agent to follow sometimes a random instruction from us which leads to a better policy \(\pi^{*}\). This approach is referred to as \(\epsilon\)-greedy learning. After our \(Q(s, a)\) calculations converging, we can send the optimal policy to our agent.

Till now, our agent was lucky and he lived in a stationary grid world (all objects besides the agent are static). But as soon as the other objects start moving the grid world gets more complex and the number of states in the set \(S\) grows enormously. This could lead to a situation where it is not possible to calculate all the \(Q(s, a)\) anymore. In such situations, we have to approximate the \(Q(s, a)\) function. One way to approximate \(Q(s,a)\) functions is to use deep reinforcement learning.

References & Sources

• Haili Song, C-C Liu, Jacques Lawarrée, and Robert W Dahlgren. Optimal electricity supply bidding by Markov decision process.IEEE transactions on power systems, 15(2):618–624, 2000.
• Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey.Journal of artificial intelligence research, 4:237–285, 1996.
• Christopher JCH Watkins and Peter Dayan. Q-learning.Machine learning, 8(3-4):279–292, 1992.
• Drunk guy image from flaticon.com
• Hombre malo image from flaticon.com
• Home image from flaticon.com
• Police image from flaticon.com