[PROJ 6] A BCI-BASED IOTs APPLICATION

Hi everybody, how are you doing?

What I am gonna present today is another project of mine on Brain/Human-Computer Interface (abbreviated as BCI). In this topic, I will not only introduce my project but also talk a little bit more about the fundamental (like what it is, how it works, and why I chose it as a method for my project).

WHAT IT IS AND HOW IT WORK?

If you watch TV, follow some Tech pages on the Internet, or read Tech news every day, you will see that BCI systems, time by time, have proved its ability to bring significant benefits to daily-life applications, especially in supporting handicapped people as an alternative communication system. There are two types of BCI that are frequently used for controlling in real life are motor imagery based and Steady-state visual evoked potential (abbreviated as SSVEP).

Motor imagery based BCI

The operation of the former one is based on the characteristics of our sensorimotor area.

To illustrate how it could be used in BCI system, let’s take a look at the video above. At first, we need to know what we are watching. The two images on the top show the mapping of C3 (left) and C4 (right) electrodes on the scalp. Now, look at the two graphs below, it describes the change of the power of the alpha rhythm (at C3 and C4) when an event-related potential (ERP) occurs (left-hand or right-hand imagery movement in this case). There is one more point that you need to notice is the relation between the top mappings and the bottom graphs. If the power of alpha at the electrodes increase, the color of the area around the electrode position will turn to more red, or else, it will turn bluer. In the next step, let’s me explain the phenomenon that is the fundamental of motor imagery based BCI system. Now, let’s take a look at the left part, when the left-hand imagery movement occurs, as we can see in the left-bottom graph, the power of alpha at the position C4 decrease significantly. In fact, concurrently, the power of beta at the position C4 increase considerably, however, it is not illustrated in the video. By looking at the mapping on the right-top of the video, we can see the right area turn to the background color whilst the left are still remained its one. Then, we could ask ourselves, why the left-hand imagery movement affect the power of alpha and beta on the right? It can be explained by the mechanism of the crossover connection in the brain (as shown in Fig. 1).

1887_28_36-right-hand-left-hemisphere

Fig. 1 The crossover connections of left hand to right hemisphere, right hand to left hemisphere [1]

There are many BCI systems were built based on this approach and many research were conducted over the world for the purpose of improving the performance and increasing the accuracy (or the classification result) of the motor imagery based BCI system. However, non-invasive BCI systems based on motor imagery is still not reliable and robust to apply to multi-task (i.e. three or more) in daily-life apps. Recently, some research groups worldwide have developed some invasive one though, it still not possible to apply to the real life because of the cost and risks of brain surgery.

SSVEP based BCI

Regarding the latter, EEG signal used in this type of BCI generated from visual cortex (Fig. 2) when it reacts to visual stimulation (such as a flickering LED at specific frequencies).

brain-anatomy-38-728

Fig. 2 Visual cortex area [2]

The principle of this type of BCI is that when you’re looking at a flickering sign or LED at a specific frequency, your brain will generate an electrical activity that has the same (or multiple of) frequency of visual stimulation [3]. For example, if we look at an 8 Hz visual stimulation (as I’ve said before, it could be a flickering LED), your brain will generate steady-state potentials at 8, 16, and 24 Hz (Fig. 3).

image10

Fig. 3 Response of brain to visual stimulation of 8 Hz [4]

Here is another video that shows an application of SSVEP based BCI in controlling a game.

We can see that, in a BCI system based on SSVEP, there are more than one visual stimulations could be used to adapt to the design condition. Each visual one is assigned to a state or control command that can be classified by using some popular classifiers (random forest, support vector machine or neural network).

The signal generated by using the SSVEP approach has a high signal-to-noise ratio and relative immunity to artifacts (like EMG, ECG, or EOG). As a result, it could be used in real-life applications reliably and robustly. However, though it could cope the disadvantage of the former one (i.e. reliability and robustness in multi-task), it can cause some annoyance to users because of exposure to flickering LED for a long time of use.

MY BCI-BASED IOTs APPLICATION

In my project, I proposed a simple approach for BCI application by combining it with IoTs. The project is implemented for the purpose of supporting the handicapped people, or more exactly, supporting families with disabled people in caring their relatives who have disabilities. The Fig. 4 shows my overall BCI system.

proj6_overallBCIs

Fig. 4 Overall BCI-based IoTs application

In the project, I used EPOC that is an inexpensive EEG device to acquire the signal from the user’s scalp. Then, EEG signal is translated to control commands by the BCI module. These commands will be received by IoTs module after that to help the handicapped people signal to their relative (as Fig. 5(a)) or send an SOS message (in case of nobody is at home as Fig. 5(b)) to show that they are in urgent need of help.

proj6_application

Fig. 5 The use of our IoTs application

To help the user to use our application easily, we design a GUI that is shown in Fig. 6.

proj6_gui

Fig. 6 Graphical user interface

And then, we build a prototype for the experiment.

proj6_prototype

Fig. 7 The prototype of the IoTs module

WHY DID I CHOOSE FOR MY PROJECT?

In my project, I decided to build the BCI system based on motor imagery because of two main reasons. Firstly, as I said before, even though the SSVEP approach is more reliable to use in daily life, it could cause some eye problems if it is used day by day. Secondly, using the BCI system based on motor imagery seems to be more natural than that based on SSVEP. The reason of that is, normally, before getting stroke or injuries, people often use their arms to reach their purpose, for example, they want to take a cup to make tea. Now, after accidents, people could also think of their limbs too, but to signal to get help from the others instead.

DEVELOPMENTS IN FUTURE

proj6_development

Fig. 8 Development in future

For further developments in future, it would be more effective if we can embed the ability of emotion recognition. On the other hand, the families with the disabled would also need to coordinate with medical organizations to support for taking care the disabled better.

If you are interested in my project, please feel free to contact me. I will be glad to share ideas and receive feedback from you.

 

Thank you and see you,

Curious Chick

 

References

[1] https://www.joshuanava.biz/memories/the-double-brain.html

[2] https://www.slideshare.net/lasitham/brain-anatomy-12098255

[3] http://synaptitude.me/blog/a-quick-intro-to-ssvep-steady-state-visually-evoked-potential/

[4] https://www.intechopen.com/books/advances-in-robot-navigation/brain-actuated-control-of-robot-navigation

 

Advertisements

[PyTUT 7] FUNCTIONS IN PYTHON

Hi everyone,

So far, we have known how to use basic control structures to perform sequential, looping and selecting tasks. In this tutorial, to take a further step, I would show you the way to help you to manage your program easier when the number of lines in your code become larger and to make the program more concise.

The first thing I wanna show you is functions. It can be seen as blocks of code written to execute a specific task. A function can return whether values (like numbers, strings, lists, etc.) by using the command return or nothing (for example, functions that are used just to display information).

How to create a function

To create a function, we use the structure below:

def name_of_the_function(parameters):                      (1)
   statements                                              (2)

   (return value)                                          (3)

Remember, just like when using for-looping structures or if-structures, we need an indentation to make the Python interpreter understand which parts of code belong to the function.

Elements of a function

OK, let’s take a look at the structure of the function above.

In the line (1), the command def help the interpreter know that you are defining a function, followed by the name of the function. In addition, a function could have parameters or not. In the next section, I will show you more detail about how to pass arguments into a function.

In the line (2), we put statements here to perform desired tasks.

The line (3) shows that a function could return a value or not, depending your purpose when writing it.

Together, let’s make a ‘Hello World’ program, but now, we will write a function to do it instead of only using the command print.

The say_hello above is a function that doesn’t return any values, its purpose is just to print the line “Hello World” on the screen.

Here is another example, in which, a function return a value of an addition.

Or you can even return a list.

All the codes above can be downloaded from my github: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut7_functions/lab1_elements_of_a_function.py.

Parameters of a function

Now, it’s time to take a further detail in parameters in function and how to pass arguments into it.

Now, let’s tweak the ‘Hello world’ program above by adding a parameter ‘name’ to it. Our program now won’t say hello world when we call it but say hello to the name that we have entered, instead.

By that way, we added a parameter and passed argument to our function.

The code can be downloaded from my github: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut7_functions/lab2_say_hello.py.

Passing a fixed number of arguments

Let’s take a look at another example below. The code show a function using Euclid algorithm to find the greatest common divisor of two nonnegative m and n.

Functions that we made above is kind of functions with fixed number of parameters. Because of this, be careful when calling a function with parameters, the number of arguments you pass need to be equal to the number of parameters, otherwise, errors will occur.

You can download the code from my github: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut7_functions/lab3_euclid_gcd.py.

Passing an arbitrary number of arguments

So, to have a function that permit you to pass the arbitrary number of arguments, there is two ways.

The first one is to put all values into a list and pass the list to the function.

The second is to put the symbol ‘*’ before the parameters of the function.

The codes can be downloaded from the link: .

Using default values

Sometimes, you want to set the default value for the parameters.

For example, back to the ‘Hello World’ example above. Now, we set the default value of parameter ‘name’ of the function to ‘World’, so, if we don’t pass any value into the function when calling it, it will print ‘Hello World!’ again, otherwise, it will say hello to the name we have passed.

The code can be downloaded here: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut7_functions/lab5_say_hello_2.py.

The scope of variables

The last thing in this post I wanna tell you is that how vital to know the scope of variables.

Let’s continue to learn from codes together.

You can download the code here: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut7_functions/lab6_scope_of_variable_1.py.

So, you can see the results clearly. The variables a and b which are initialized in the function check_print just exist while the function was running.

Now, if you wanna use the variables a and b as global variables, how could you do that? What if we don’t declare variable names in the function like the code below?

It’s OK, but don’t do that frequently, it sometimes brings to you serious mistakes in your code. To use global variables, I prefer to put ‘global’ before the name of variable in the function.

The code can be downloaded here: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut7_functions/lab7_scope_of_variable_2.py.

OK, that’s all I wanna talk about in this post, in the next tutorial, I will introduce basic of classes in Python.

Hope you enjoy it,

Curious Chick

[PyTUT 6] CONTROL STRUCTURES

Hi everyone!

First of all, I would like to say thank you to all my friends and people who support me to make this series of Python tutorial. Two weeks ago, I shared the series with some of my friends and got many positive comments from them and they encouraged me to continue further. I would say that your concern really is my motivation to continue to do this.

OK now, let’s take a look at last tutorials a bit, in which, I introduced how to do some basic math with numbers, play with strings and three types of Python array, then assign values to variables.

To take a further step, in this tutorial, I want to show you control structures in Python.

By the way, I set up a new account on Github and from now, all codes will be published to the link below.

https://github.com/tenga061/basic_python_tutorials

And all codes in this tutorials were published on:

https://github.com/tenga061/basic_python_tutorials/tree/master/pytut6_control_structure

As usual, I recommend you to read the following topics before beginning this tutorial.

Related topics

Introduction

There are three types of control structures as you can see in Fig. 1. The first one is sequence structure that you used in the preceding tutorials. The second and last one is selection and loop structure, respectively.

fig.1

Fig. 1. Flow charts of three types of control structures [1]

In most programs, the two last structures proved to be helpful to make the codes more concise or more clear. But now, before beginning with them, let’s try using sequence structure again to make a simple program that print numbers from 0 to 9 on the screen but using Python script editor instead of Python Shell (or Python interpreter prompt).

Sequence

To open Python script editing window, do the following steps:

  • First, you need to open IDLE (read [PyTUT 1] AN INTRODUCTION OF PYTHON again if you don’t remember how to open IDLE).
  • Second, press CTRL + N  or left click on File -> New File on IDLE tool bar in order to open the Python script window.

fig.2_1

Fig. 2 Open IDLE script
  • Third, in Python script editing window, press CTRL + S or left click on File -> Save and name the file to save the script. Actually, it is unnecessary to save the script when you begin to code, but you ultimately must do this if you want to run the script.

fig.3_1.png

Fig. 3 Save script file

Following to this, we will start to write a first script that print the numbers from 0 to 9.

print 'The number from 0 to 9'
print 0
print 1
print 2
print 3
print 4
print 5
print 6
print 7
print 8
print 9

After that, press F5, then select OK when you are asked whether you wanna save this program or not in order to run the script (see Fig. 4).

fig.4_1

Fig. 4 Save all changes in your code before running

The result is showed in Fig. 5.

fig.5_1.png

Fig. 5 The result was showed on Python Shell

This code is available on: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut6_control_structure/lab1_sequence_structure.py.

As you can see, in sequence structure, the computer will start to run the program from the begin to the end and follow the statements in order. However, imagine what if you want to print the numbers from 0 to 10,000. OK, it is still possible if you have a lot of time, right? Believe me, you will not want to do this.

For the reason of this, in the next section, I will introduce to you repetition structures which will help you to solve such repetitive problems.

Repetition

As I mentioned before, repetition structures are useful for repetitive problems. There are three types of repetition structures, including: for-loop, while-loop, and do-while-loop. First, look at the for-looping structure as showed in Fig. 6.

For-Looping Structure

The flow chart in Fig. 6 was taken from a C++ book but I think it’s still good to help you understand the mechanism of the for-looping structure in Python or even in any other programming languagues you want to learn in the future.

fig.6

Fig. 6 For-Looping Structure [1]

In C++, to create a for-loop, you first need a initial statement to initialize a loop variable. Then, declare a loop condition such as i < 10, where i is a loop variable. If i satisfies the condition (i < 10), statements in the loop will be performed and the loop variable will be updated after that (like i = i + 1 or i = i + 3, etc.). By contrast, the computer will close the loop and perform another statement.

The mechanism of for-loop in Python is similar to C++. However, if you were not familiar with C++ before, it still is OK. Let’s take a look at the example below that print numbers from 0 to 10 using the for-looping structure in Python:

for i in range(0, 10):
   print i

As you can see, to create a for-loop in Python, we first initialize a loop variable i, just like C++. However, there is a little bit different from C++. In Python, to create a loop condition, we use range(m, n) function which make a list from m to (n-1) with n>m, the upper bound n of the range function is implied as the loop condition i < (n-1). Additionally, the loop variable will automatically update itself in every value in the list in every loop (see Fig. 7 to understand how it works).

fig.7

Fig. 7 The way the for-loop structure work in Python

Now, what if you want to print only odd numbers from 0 to 9? Fortunately, you can add step argument into the range(m, n, step) function. So, the code become:

for i in range(1, 10, 2):
   print i

(This script is available on: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut6_control_structure/lab2_for_loop.py)

The results are illustrated in Fig. 8.

fig.8

Fig. 8 For-looping structure example results

Not only those, for-looping structure of Python is even more flexible, by which, you can loop to print arbitrary numbers as you want by assigning the value of each element in the list to loop variable.

arbitrary_numbers = [9, 123, 59, 1039, 23, 27]
for number in arbitrary_numbers:
   print number

You can even do this for tupe:

lovely_cities = ('Paris', 'Los Angeles', 'Seoul', 'Tokyo')
print 'Here is some of the lovely cities: '
for city in lovely_cities:
   print '   + ' + city

Or dictionary type:

hp_characters = {'Danial Radcliffe': 'Harry Potter', 
                 'Emma Watson': 'Hermione Granger',
                 'Rupert Grint': 'Ron Weasley'}
for actor, character in hp_characters.iteritems():
   print actor + ' played ' + character + ' in the series of Harry Potter'

OK, you saw how useful is it when using for-loop to solve repetitive tasks. But you can only use the for-loop if you know exactly how many time it loop. For some problems that you don’t now the number of loop, you need to use while-looping structures which will be introduced in the next section.

While-Looping Structure

fig.4.png

Fig. 9 While Looping Structure [1]

As I mentioned before, in while-loop, you don’t needa know how many times your loop will run but loop conditions such as i \leq n or i \neq n . In the case, we use while-looping structure.

While-looping structure begin by checking the condition loop or expression like i \leq n . Then, if the condition is satisfied, statements in the loop will be executed. Otherwise, the loop will be ended. Note that, of course we can’t type symbol like \geq or \neq in any programming languages. Instead, we use these signs in the table below.

Table 1. Relational operators [1]

operator_in_python

For example, we want to write a program that read arbitrary numbers from user. If the number is equal or less than 9, the program is continue, by contrary, it stop and print the line “Goodbye and see ya”.

user = 1

while (user<=9):
   a = raw_input('Please enter an arbitrary number: ')
   user = float(a)

print 'Goodbye and see ya'

The code can be downloaded here: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut6_control_structure/lab3_while_loop_game_1.py.

Now, what if you wanna use more than one looping condition, for example, the program will continue requiring you to enter an arbitrary number if your entered-number is equal or less than 9 and more than 6. In the case, we needa use boolean operator and.

user = 1

while (user>6) and (user<=9):
 a = raw_input('Please enter an arbitrary number: ')
 user = float(a)

print 'Goodbye and see ya'

Similarly, you can use boolean operator or in Python by typing or. It’s little bit different for people who were familiar with C++ or other lower programming language, in which, they must use && for the boolean and and || for the or. So, please note that, in Python, we just needa type and for the and and or for the or.

Do-While-Looping Structure

fig.5

Fig. 10 Do-While-Looping Structure [1]

The only difference between while-loop and do-while loop is that statments in the loop will be excuted at least one. In detail, if the looping condition is wrong from the begining of the loop, the while-loop will end without any statements executed, while statements in do-while-loop still is executed once because it is executed before looping conditions are checked.

Nested Looping Control Structures

Just like other programming languages, we can use looping structures in looping structures to handle more complicated tasks. You can write a simple code that paint a Christmas tree using only for loop. Here is my sample:

for m in range(3):
   for i in range(1, 5+m):
      for j in range(5-i+1):
         print ' ',
      for k in range(5-i+1, 5+i):
         print '*',
      print

And the result:

christmas_tree

Fig. 11 The result of the Christmas tree program

You can download the code from my github link: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut6_control_structure/lab4_christmas_tree.py.

Python loop controls

In this part, I would like to introduce some special commands that help you handle your loop somtimes.

pass

As its name, it mean ‘don’t do anything’. It sounds pointless, but sometimes we use it with the loop to wait for an occurrence of an event or an interupt.

Here is an example of using pass command in a loop:

while(1):
   pass

The program above will run until you close the termial.

break

This command is another way to end looping structures whenever it satisfy the condition that you set.

For example,

i = 0
while 1:
   print i
   i = i + 1
   if i > 5:
      break

Running the program and you will get the result as illustrated in Fig. 12.

Theoretically, the program should run and print the value of i till we close the terminal. However, because we set a break in the if block (will be introduced in the next section), the break break the loop when i = 6.

break_command

Fig. 12 An example of using break for while-loop

You can also use the break command in for-loop. The example below using for-loop but show the same result.

for i in range(10):
   if i > 5:
      break
   print i

When I was an undergraduate student, my professor who taught me the subject ‘Programming Fundamentals’ often prohibit us from using this command because, as he said, the break break the structure of programs. However, I think this control command sometimes is useful, especially, it is used frequently in programming for microcontroller. Anyway, you should consider carefully if you want to use it in your program.

continue

This command returns the control to the beginning of the loop. Let’s run the following program to see how it work.

i = 0
while i<10:  
   i = i + 1
   if i == 5:
      continue
   print i

The result is described in Fig. 13.

continue_command

Fig. 13 An example of using the command continue in a program

We can see that the command continue in the program above is the reason why we miss the number 5. In details, when i reach the value 5, the if condition is satisfied, then the command continue is run. As a result, the control of the loop go back to the begining without execute the line print i (i.e print i = 5 on the screen).

Selection

Now, we turn to the last structure of this tutorial – selection or if-structure. In the previous section (Python loop control), I briefly show you using the selection structure to make condition for breaking the loop. Now, we introduce further.

One-Way Selection

The simplest form of selection structures are one-way selection. In which, we use an if condition or expression to decide whether statements should be executed or not. In selection structures, the expression is described as the same way as the expression of while-loop in Table 1.

fig.2

Fig. 14 One-Way selection [1]

Let’s me show you an example in the normal life. Assume that you go to the market and want to buy a fish for lunch. You go to a stall and ask the price of a fish, if it is lower than $10, you buy it, but if not, you don’t. Here is a code:

print 'How much money for this fish?'
fish_price = raw_input("It's: $")
fish_price = float(fish_price)
if fish_price>10:
   print 'Sorry, I don\'t wanna buy it'

Now, run the program, enter 20 for the cost of the fish, you will get the result ‘Sorry, I don’t wanna buy it’.

Multiple Selection

fig.3

Fig. 15 Two-Way selection [1]

From the view of real life problems, you can see that it’s not simple like that. Here is another script: You go to the supermarket, ask the price of the fish, if the price less than $10 then you buy it, if not, you deal with the salesman that if you could buy it with the price of $13.

To do the script above we need to know how to use the multiple selection structure. Here is the structure of two-way selection (as described as Fig. 15)

if condition:
   statements
else:
   statments_2

If you need more than 2, we use elif for conditions and else for the last one.

if condition_1:
   statements
elif condition_2:
   statements_2
...
elif condition_n:
   statments_n
else:
   statements_(n+1)

Here is the script of the problem above.

print 'How much money for this fish?'
fish_price = raw_input("It's: $")
fish_price = float(fish_price)
if fish_price<10:
   print 'OK, give me one.'
else:
   print 'Could I buy it with $13?'

Nested Selecting Control Structures

Just like repetition structures, you can put selection structures into others. Here is a small example of using nested selecting control structures.

Just for fun

OK, so, I introduced you the foundation of control structures in Python. Now we can use it to make a simple entering-password code. In which, users are asked to enter their password, if the entered password is right, the program will show them where were  the mythical treasures hide, if not, the program will end with curse: ‘You have wrong answer for 5 times. Now, you will not be able to eat candy. Ha ha ha!’.

You can download the code from my github: https://github.com/tenga061/basic_python_tutorials/blob/master/pytut6_control_structure/lab5_mythical_treasure.py.

So this is the end of this topic. In the next, I would introduce you function, class and package in Python. Goodbye and see you again.

Hope you enjoy it,

Curious Chick

References

[1] D. S. Malik, C++ Programming: From problem analysis to program design (Sixth Edition), Cengage Learning (2013)

LINEAR REGRESSION (3) – SOLVING NON-LINEAR PROBLEMS

Hi everyone! It has been a long time no see. I got extremely busy preparing my application for the deadline of Spring 2018 semester admission at universities to study at a higher level of education and do my further researches on the field of Machine Learning and Human-Computer Interface. It really was a difficult process, occupied most my time for travelling around and around the city for preparing necessary paperworks, writing statments like statment of purpose or study plan and sending emails to my professors to ask for recommendation letters.

Related Topics:

Introduction:

OK, just a little talk, to begin this topic, let’s remember, I introduced you how to implement a Linear Neuron Model to solve a linear problem with the aid of famous library Tensorflow. However, remember that how frequently do you face with the linear problem? Honestly, it rarely happens, even the “Materialism” problem related to the number of girl friend and monthly salary we dealt with in the first and second topic of “Linear Regression” is also not a pure linear problem. Let me show you what is a problem you get when using linear model for this. You might say that it was very good at trying to guess the number of my may-be-have girl friends (GF) in the first topic. But how about trying to use this linear model to predict the number of girl friends of the richest man in the world – Bill Gates (OK, this is just a joke, I mean no offence when I said that). From [1], we know Bill’s monthly income is \$216,666,666.00 , infer from the model, we get the number of GF he may have is 0.45753154 + 0.00087198*216,666,666.00 = 188,929.457 . How could it be real? Theoretically, may be. Practically, it’s infeasible, even Casanova couldn’t reach this number. Now, you see the problem here, if the monthly income of a man increase to infinite, the number of GF he has will increase to infinte too while the limit on the number of population worldwide or others such as morality, religion, etc. make it impossible. To deal with this, we need a restriction on the upper bound.

Dataset

First, we needa add a little bit more in our dataset to make a restriction. Assume that, even though there is a man who earn more than $15,000 monthly, the maximum number of GF he has are not higher than 10.

training_dataset

Fig. 1 The new dataset of the relation between monthly income and the number of GF of a man

The dataset can be downloaded from my github link: https://github.com/tenga061/machine_learning/blob/master/linear_regression/episode_3/dataset/funny_dataset.csv.

For this dataset, we won’t try to guess the number of Bill’s GF because, as you know, we don’t have enough data (our dataset stop at $20,000). Instead, we will implement a polynomial model next, and then use it to test another called testing dataset.

Polynomial regression model

Polynomial function

For our dataset, the polynomial model has form:

\hat{y} = f(x, \omega) = \omega_{0} + x_{1} \omega_{1} + x_{1}^{2} \omega_{2} + ... + x_{1}^{n} \omega_{n} \indent (1)

Or we can rewrite it into:

\hat{y} = f(x, \omega) = \sum_{i=0}^{n}{x_{1}^{i} \omega_{i}}

Or by matrix form:

\hat{y} = f(x, \omega) = \bold{X} \omega \indent \indent \indent \indent (2)

Let’s look at the equation (1), f(x, \omega) is a nonlinear function of x though, it is a linear function of the coefficient \omega . Because of that, we can still implement the polynomial model as the same way as we do it with the linear one in the first topic. But now, we have the form of the matrix \bold{X} is:

\bold{X} = \begin{bmatrix} 1&x_{1}&x_{1}^{2}&...&x_{1}^{n}\\ 1&x_{2}&x_{2}^{2}&...&x_{2}^{n}\\..\\1&x_{k}&x_{k}^{2}&...&x_{k}^{n} \end{bmatrix}

Implementing polynomial model using TensorFlow

To form input vector after feeding it into the model, I wrote the function below.

Forming X data:

# Creating the input matrix X
def create_x(order):
   global mans_monthly_income

   # Creating element vector
   X = np.ones((len(mans_monthly_income)), 1)
   for i in range(1, order+1):
      temp = np.power(mans_monthly_income, i)
      x_temp = np.array(temp, dtype=float)
      x_temp = np.transpose(np.matrix(x_temp))

      X = np.column_stack((X, x_temp))
   
   return X

Forming y data:

# Creating the target vector y
def create_y():
   global total_number_of_gfs
  
   y = np.array(total_number_of_gfs, dtype=float)
   y = np.transpose(np.matrix(y))

   return y

The results are showed below when using polynomial function with order of 3, 7 and 9, respectively:

polynomial_order_3

Fig. 2 The model with order of 3

polynomial_order_7

Fig. 3 The mode with order of 7

polynomial_order_9

Fig. 4 The model with order of 9

The code can be downloaded from my github: https://github.com/tenga061/machine_learning/blob/master/linear_regression/episode_3/code/12_lr_nonlinear_polynomial.py.

Testing the model

The question now is that how could we know which order is the best? To answer this, I made another dataset (red points in Fig. 5) for the purpose of testing our model by evaluating the root-mean-square (RMS) error (showed as Eq. (3)) each order from 0 to 10.

E_{RMS} = \sqrt{\frac{1} {N} \sum_{i=1}^{N} {(\hat{y} - y)^{2}}} \indent \indent \indent \indent (3)

test_dataset

Fig. 5 Training (blue points) and testing (red points) dataset

The testing dataset can be downloaded here: https://github.com/tenga061/machine_learning/blob/master/linear_regression/episode_3/dataset/funny_dataset_test.csv.

rms_error_graph

Fig. 6 The relation between the value of RMS error and the order of polynomial function

The code can be downloaded from my github: https://github.com/tenga061/machine_learning/blob/master/linear_regression/episode_3/code/13_lr_nonlinear_polynomial_rms.py.

From the Fig. 6, we can see that the RMS value of models with order higher than 8 begin increasing. In that case, we say our model was overfitted. By which, the model couldn’t work well with other test datasets (even it may work wrong), though it have done well with the training. To deal with this problem, there is some techniques can be used.

Overfitting

Increasing the size of the dataset

The first is to collect more and more data. This depends on the data acquisition step, sometimes we can, but sometimes not. There is a ‘lex non scripta’ supported by many is that the number of data points should be larger than some multiple (from 5 to 10 is OK) of the number of parameters in the model.

Regularization

However, what happens if you are in the situation that you cannot acquire more data. In the case, I would like to introduce you a simple technique of regularization by adding \frac{1} {2} \lambda \omega^{2} to the loss function \mathcal{L} (\omega) .  Now, the loss function become:

\mathcal{L}(\omega) = \frac{1} {2} \sum_{i=1}^{N} {(y_{i} - \bold{x}_{i})^{2} \omega} + \frac{1} {2} \lambda \omega^{2} \indent \indent \indent \indent (4)

The lambda in the adding element help us penalize the loss when the model become overfitting and keep the parameters small.

Turn Eq. (4) to the matrix form:

\mathcal{L}(\omega) = \frac{1} {2} (\bold{y} - \bold{X} \omega)^{2} + \frac{1} {2} \lambda \omega^{2} \indent \indent \indent \indent (5)

Calculating the derivation of Eq. (5) and set it equal 0 to find roots of the model:

\frac{\partial{\mathcal{L} (\omega)}} {\partial{\omega}} = -\bold{X}^{T} (\bold{y} - \bold{X} \omega) + \lambda \omega = 0

Similarly,

(\bold{X}^{T} \bold{X} + \lambda \bold{I}) \omega = \bold{X}^{T} \bold{y}

Assume that (\bold{X}^{T} \bold{X} + \lambda \bold{I}) is a nonsingular matrix, we have parameters of model is the root of the equation:

\omega = (\bold{X}^{T} \bold{X} + \lambda \bold{I})^{-1} \bold{X}^{T} \bold{y}

Implementing polynomial with regularization using Scikit-Learn lib

The part of code to implement the model with regularization using Scikit-Learn is showed below:

# Creating the model and feeding data to it
model = linear_model.Ridge(alpha=lamda)
model.fit(X, y)

# Getting the coefficients
coef = model.coef_
params = coef[0]
print params

The code can also be downloaded from my github: https://github.com/tenga061/machine_learning/blob/master/linear_regression/episode_3/code/16_lr_regularization_sklearn.py.

Implementting polynomial with regularization using TensorFlow lib

The part of code to implement the model with regularization using TensorFlow is showed below:

# Creating constant opt to store data
X_tensor = tf.constant(X)
y_tensor = tf.constant(y)

''' Finding the root of the model '''
# Calculating Xt*X
first_step_1 = tf.matmul(tf.transpose(X_tensor), X_tensor)
# Calculating lamda.I
temp_step = (order+1)*lamda*np.identity(order+1)
first_step_2 = tf.constant(temp_step)
# Calculating Xt*X + lamda.I
first_step = tf.add(first_step_1, first_step_2)
# Calculating (Xt*X + lamda.I)^(-1)
second_step = tf.matrix_inverse(first_step)
# Calculating (Xt*X + lamda.I)^(-1)*Xt
third_step = tf.matmul(second_step, tf.transpose(X_tensor))
# Calculating (Xt*X + 2.lamda.I)^(-1)*Xt*y
root = tf.matmul(third_step, y_tensor)

''' Getting parameters '''
with tf.Session() as sess:
   params = sess.run(root)

print params

The code can also be downloaded from my github: https://github.com/tenga061/machine_learning/blob/master/linear_regression/episode_3/code/15_lr_regularization.py.

Sigmoid function

Another way to confront with nonlinear problems is to use functions having features suiting dataset features. This is based on your experience much.

Look at our dataset, on the one hand, we can see points distributing in a curve that is partly look like a function called “sigmoid” (as showed as Fig. 7).

sigmoid_function

Fig. 7 The graph of sigmoid function

On the other hand, we have a restriction on the upper bound, so it can be easily recognised that these features suit sigmoid’s features well. For the reason, now, we will use the model showed in Eq. (6) instead of the polynomial model as before.

\hat{y} = f(x, \omega) = \omega_{0} + \frac{1} {1 + e^{-x}} \omega_{1} \indent \indent \indent \indent (6)

However, note that the input x is very large in our dataset (up to 20,000), so e^{-x} is very small and can be set as 0. For that reason, we need to add a trick to the Eq. (6) by dividing x by a constant number k. In this case, I use k = 4000, the Eq. (6) turn into:

\hat{y} = f(x, \omega) = \omega_{0} + \frac{1} {1 + e^{-x/4000}} \omega_{1} \indent \indent \indent \indent (7)

The new model is illustrated in Fig. 8.

sigmoid_function

Fig. 8 The sigmoid model

The code can be downloaded from my github: https://github.com/tenga061/machine_learning/blob/master/linear_regression/episode_3/code/9_lr_nonlinear.py.

Apart from polynomial or sigmoid function, we can also apply others like trigonometic or logaric function for our regression model.

So, that’s all I want to talk about how to use linear model for regression tasks. In the next topic, I would introduce the way to use linear model, but for classification tasks.

Hope you enjoy it, goodbye, and see ya,

Curious Chick

References

[1] https://paywizard.org/main/salary/vip-check/world-richest-people

[2] Simon Rogers, Mark Girolami – A first course in Machine Learning

[3] Christopher M. Bishop – Pattern Recognition and Machine Learning

[4] http://scikit-learn.org/stable/modules/linear_model.html

[PROJ 5] BRAIN COMPUTER INTERFACE

Many people who watch sci-fi film a lot may be familiar with the idea of using brain to control devices or even people. In fact, this have been partly done thanks to developments in neuroscience and computational methods. Of course, till now, scientists have not yet pioneered any technology which can be able to be used to control other people. But at least, Brain-Computer interface (BCI) – a recently developed technology relating to the classification of brain waves – helps us to control some electrical and electronic devices, which bring sharp benefits to people, especially the disabled one. For example, the disabled can use BCI to activate their wheel chair or switch some household devices like TV, air conditioner, etc. without any help from others. On the other hand, BCI technology can also be used for educational purposes or analyzing decision making of customers in fields of economy such as sale and marketing, too name but a few. From these, it can be said that, BCI has been becoming one of sexiest disciplines in the world, along with machine learning and internet-of-things trends.

Limitations notwithstanding, BCI still is an awesome technology in the belief that, by which, we can make superpowers real or become superheroes in the near future. With this belief, I tried to convince my professor to allow me to build a non-invasive BCI system for my thesis, and after a while of deliberation, he agreed. The result is showed in the video below.

 

From an old story,

Curious Chick

[PROJ 4] LINE FOLLOWING ROBOT

This is our project on the 8th semester which belongs to the subject “Mechatronic system desgin”.

Honestly, it was the first time the mobile robot achieved the line, so it is kind of slow (take about 40 seconds to accomplish).

However, the final result was nearly 15 seconds which is only after the top of my class (approximately 14 seconds).

From an old story,

Curious Chick

LINEAR REGRESSION (2) – NEURAL NETWORK APPROACH

Related Topics

Introduction

Hi everybody,

In the last post, I depicted how to implement linear regression based on inverse matrix approach or normal equation using both Scikit-learn and TensorFlow libs. Continue reading “LINEAR REGRESSION (2) – NEURAL NETWORK APPROACH”