Sign Language Coach (AVR)

Sign Language Coach

Our motivation is two-fold. Aside from helping deaf people communicate more easily, the SLC also teaches people to learn the ASL. Our product, a sign language coach (SLC), has two modes of operation: Teach and Learn. The SLC uses a glove to recognize the hand positions and outputs the ASL onto an LCD. The glove detects the positions of each finger by monitoring the bending of the flex sensor. Below is a summary of what we did and why:

1) Build flex sensor circuit for each finger. Sew flex sensors and accelerometer onto glove to more accurately detect the bending and movement of thecomponents.

2) Send sensor circuit output to MCU A/D converter to parse the finger positions.

3) Implement Teach mode. In Teach mode, the user “teaches” the MCU ASL using hand gestures. To prevent data corruption, A/D converter output and the associated user specified alphabet are saved to eeprom, which can only be reset by reprogramming the chip.

4) Implement LEARN mode. In Learn mode, the MCU randomly chooses a letter it has been taught and teaches it to the user. The user ”learns” by matching his hand positions to that which the MCU associated with the letter. Using the LCD, he can adjust his finger positions appropriately. The finger positions are matched to the appropriate ASL using an efficient matching algorithm.

Expectations

Our original expectations for the Sign Language Coach unit outlined in our proposal are listed below:

A. Basic Functionalities

§ Accurately detect finger movements.

§ Recognize sign language of the most frequently used letters of the alphabet such as {a,e,i,o,u,r,s,t,l,n}.

§ Designate a hand position that indicates the transition from end of the current letter to the begining of the next letter.

B. Enhancements (Time-Permitting)

§ Extend sign language to the entire alphabet. This will require the addition of accelerometers.

§ Recognize the transition from the current letter to the next letter without an indicative hand position.

C. Complex Features (Time-Permitting)

§ Concatenate sign language letters to speak words and sentences.

This project met our expectations. It accomplishes more than what we initially planned for it to do, including being able to recognize all the letters of the alphabet. Still, there are many add-ons that are possible for our project. We could have done many things differently and can do a lot to improve our product. Some improvements include implementing a free mode in addition to the existing train and program modes, where the system will recognize any letter being signed. We also limited our project to the alphabet; implementing word vocabulary would have been a great addition to the project. We could also have improved our unit by adding sound. Using a more powerful microcontroller with more memory was required for the system to successfully “speakthe signed letters. Our small 16 by 4 LCD also provided some limitations; a larger LCD would have been useful for creating a more user-friendly display. Another improvement could have been made to the physical connections of the flex sensors to the microcontroller. The long wires are very cumbersome, and the use of cable wire would have looked cleaner. However, given the time constraint that we had and the limitations of the Atmel 32, we believe our project to be a success

Link: http://instruct1.cit.cornell.edu/courses/e...asl1/index.html

CODE

// A to D test code
// NOTE -- You MUST MOUNT the Aref jumper

#include <Mega32.h>
#include <stdio.h>
#include <stdlib.h>
#include <delay.h>
#include <math.h>
#define LCDwidth 16 //characters
#define t1 10 //
#define t2 1000
#define t3 10
#define t4 1000 //runs every second (1000ms)

//I like these definitions
#define begin {
#define end }
#define NOTFOUND -1
#define RANGE 5 //accepted finger position interval

//State machine state names
#define NoPush 1
#define MaybePush 2
#define Pushed 3
#define MaybeNoPush 4
#define LEARN 1
#define TEACH 0
#define let_a 97
#define let_z 122
#define let_A 65
#define let_Z 90
#define sensitivity 3

#asm
.equ __lcd_port=0x15
#endasm
#include <lcd.h> // LCD driver routines

char lcd_buffer[17], time1; // LCD display buffer

typedef enum hand {thumb, index, middle, ring, pinky} finger;
typedef enum boolean {FALSE, TRUE} bool;
typedef enum alphabet {A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z, NONE} letter;
const char letters[] = {'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','--'};
char mode, dir[5]={'+','+','+','+','+'};
int percent[5]={0,0,0,0,0};
int index_set[26], N=0;

//function declarations
int match(int pos[]);
void task1(void); //test for button press
void task2(void); //increment note to be played
void task3(void); //button debouncer
void task4(void); //delay 3seconds
void gets_int(void); //starts getting a string from serial line
void puts_int(void); //starts a send to serial line
void set_values(letter l, int arr[]);
void calc_percent(int my_letter, int pos[5]);
void plusminus(void);

//variable declarations
eeprom int alpha[26][5];
char Ain, PushFlag, PushState, let='-'; //raw A to D number
int index_found=1, my_index=-1;

unsigned int time2, time3, time4=0; //task scheduling timeout counters
unsigned long time;
unsigned int v;

//RXC ISR variables
unsigned char r_index; //current string index
unsigned char r_buffer[16]; //input string
unsigned char r_ready; //flag for receive done
unsigned char r_char; //current character

//TX empth ISR variables
unsigned char t_index; //current string index
unsigned char t_buffer[16]; //output string
unsigned char t_ready; //flag for transmit done
unsigned char t_char; //current character

//**********************************************************
//timer 0 overflow ISR
interrupt [TIM0_COMP] void timer0_overflow(void)
begin
time++;
//Decrement the three times if they are not already zero
if (time1>0) --time1;
if (time2>0) --time2;
if (time3>0) --time3;
if (time4>0) --time4;
end

//**********************************************************
//UART character-ready ISR
interrupt [USART_RXC] void uart_rec(void)
begin
r_char=UDR; //get a char
UDR=r_char; //then print it
//build the input string
if (r_char != '\r') r_buffer[r_index++]=r_char;
else
begin
putchar('\n'); //use putchar to avoid overwrite
r_buffer[r_index]=0x00; //zero terminate
r_ready=1; //signal cmd processor
UCSRB.7=0; //stop rec ISR
end
end

/**********************************************************/
//UART xmit-empty ISR
interrupt [USART_DRE] void uart_send(void)
begin
t_char = t_buffer[++t_index];
if (t_char == 0)
begin
UCSRB.5=0; //kill isr
t_ready=1; //transmit done
end
else UDR = t_char; //send the char
end
/**********************************************************/

//**********************************************************
//Task 1 input a string and print it
void task1(void)
begin

char i,j, k=0;
int pos[6][5];
int sumpos[5];
int avgpos[5];
int pos1[5];
char flag;
int diff=0;
int per1, per2, per3, per4, per5;
int move[7] = {0,0,0,0,0,0,0};

int my_letter;
time1=t1; //reset the task timer
//print ad get another serial string

if (r_ready && ~PIND.7)
begin
mode = TEACH;
sscanf(r_buffer,"%d",&v);
gets_int();

let = r_buffer[0];
lcd_gotoxy(0,3); //position to upper left on display
//convert to upper case
if (let>=let_a && let<=let_z) let = let -32;
else if (let>=let_A && let<=let_Z);
else
begin
lcd_gotoxy(0,3); //position to upper left on display
lcd_putsf("INVALID LETTER"); //string from flash
printf("invalid\n\r");
return; //exit if invalid
end

printf("letter %c\n\r", let);

lcd_clear();
lcd_gotoxy(4,0); //position to upper left on display
sprintf(lcd_buffer,"PROGRAM %c", let);
lcd_puts(lcd_buffer);
lcd_gotoxy(0,2); //position to upper left on display
lcd_putsf("SET HANDPOSITION"); //string from flash

printf("Set up your hand position...\n\r");
printf("You have 3 seconds to prepare your hand...\n\r");
task4(); //delay 1 seconds
task4(); //delay 1 seconds
task4(); //delay 1 seconds

printf("Begin calibrating...\n\r");
//parse ADC0-ACD4
for (j=0; j < 6; j++) begin
for (i=0;i<5;i++)begin
ADMUX=0x60+i;
Ain = ADCH; //get the sample
ADCSR.6=1; //start another conversion
while(ADCSR.6==1);
pos[j][i]=Ain;
end
task4();
end
printf("End Calibrating...\n\r");
lcd_gotoxy(6,3); //position to upper left on display
lcd_putsf("DONE!"); //string from flash

for (i = 0; i < 6; i++)
begin
sumpos[i] = 0;
end
for (i = 1; i < 6; i++) begin
sumpos[0] = sumpos[0] + pos[i][0];
//printf("Average pos: %d Thumb Pos: %d", sumpos[0], pos[i][0]);
sumpos[1] = sumpos[1] + pos[i][1];
sumpos[2] = sumpos[2] + pos[i][2];
sumpos[3] = sumpos[3] + pos[i][3];
sumpos[4] = sumpos[4] + pos[i][4];
end
for (i = 0; i < 5; i++)begin
avgpos[i] = floor(sumpos[i]/5);
//printf("Average pos T: %d", avgpos[0]);
end

/* printf("Av T: %d\n\r",avgpos[0]);
printf("Av I: %d\n\r",avgpos[1]);
printf("Av M: %d\n\r",avgpos[2]);
printf("Av R: %d\n\r",avgpos[3]);
printf("Av P: %d\n\r",avgpos[4]);
*/
set_values(let-65,avgpos);
printf("learned %c\n\r", letters[let-65]);

end //if (r_ready && PIND.7)

else if (PIND.7)
begin
mode = LEARN;

//pick a random letter to test user
//generate number between 0 and 25
//MAKE SURE TEACH before LEARN

if (index_found == 1 && N!=0)
begin
if (N > 25)
begin
my_index = index_set[rand()/(int)(((unsigned)RAND_MAX + 1) / 26)];
index_found=0;
end
else
begin
my_index = index_set[rand()/(int)(((unsigned)RAND_MAX + 1) / N)];
index_found=0;
//printf("index is %d\n\r", my_index);
end
end
if ((~PIND.7) || (N==0))
begin
my_index=26;
end

//parse ADC0-ACD4
for (j=0;j<7;j++)
begin
for (i=0;i<7;i++)
begin

ADMUX=0x60+i;
Ain = ADCH; //get the sample
ADCSR.6=1; //start another conversion
while(ADCSR.6==1);
if (i > 0 && i < 5) begin
pos1[i]=(int)Ain;
end
else if (i == 5) begin
pos1[0] = (int)Ain;
end
else if (i == 6) begin
//movement = Ain;
move[k++] = (int)Ain;
end
end
end
diff = abs(move[4]-move[0]);

my_letter = match(pos1);
calc_percent(my_index, pos1);

plusminus();

lcd_gotoxy(0,0); //added!!
lcd_putsf(" ");

lcd_gotoxy(5,0); //position to upper left on display
sprintf(lcd_buffer,"TRY %c ", letters[my_index]);
lcd_puts(lcd_buffer);

if (my_letter != NOTFOUND)
begin
lcd_gotoxy(2,1); //position to upper left on display
sprintf(lcd_buffer,"%cT %cI %cM %cR %cP",dir[0],dir[1],dir[2],dir[3],dir[4]);
lcd_puts(lcd_buffer);
lcd_gotoxy(0,2); //position to upper left on display
lcd_putsf("%: "); //string from flash
lcd_gotoxy(0,3); //position to upper left on display
lcd_putsf(" "); //string from flash

//display matching letter
if (my_letter == my_index)
begin

//convert to percentage
lcd_gotoxy(2,1); //position to upper left on display
//change to --
sprintf(lcd_buffer,"%cT %cI %cM %cR %cP",dir[0],dir[1],dir[2],dir[3],dir[4]);
lcd_puts(lcd_buffer);
lcd_gotoxy(2,2); //position to upper left on display
per1=abs(percent[0]);
per2=abs(percent[1]);
per3=abs(percent[2]);
per4=abs(percent[3]);
per5=abs(percent[4]);
sprintf(lcd_buffer,"%2d %2d %2d %2d %2d", per1,per2,per3,per4,per5);
lcd_puts(lcd_buffer);
if (my_index == 9 && (diff> sensitivity)) //letter j needs movement
begin
lcd_gotoxy(0,3);
lcd_putsf(" MATCH! ");
task4(); //display for 1 sec
task4(); //display for 1 sec
lcd_clear();
index_found = 1;

end
else if (my_index == 25 && (diff> sensitivity))
begin
lcd_gotoxy(0,3);
lcd_putsf(" MATCH! ");
task4(); //display for 1 sec
task4(); //display for 1 sec
lcd_clear();
index_found = 1;
end
else if ((diff<= sensitivity) && (my_index != 9) &&(my_index != 25))
begin
lcd_gotoxy(0,3);
lcd_putsf(" MATCH! ");
task4(); //display for 1 sec
task4(); //display for 1 sec
lcd_clear();
index_found = 1;
end

end
/*else
begin
printf("Place 3 \r\n");
task4(); //display for 1 sec
lcd_gotoxy(2,3); //position to upper left on display
sprintf(lcd_buffer,"LOOKS LIKE %c ", letters[my_letter]);
lcd_puts(lcd_buffer);
end*/
end
else
begin
lcd_gotoxy(2,1); //position to upper left on display
sprintf(lcd_buffer,"%cT %cI %cM %cR %cP",dir[0],dir[1],dir[2],dir[3],dir[4]);
lcd_puts(lcd_buffer);

lcd_gotoxy(0,2); //position to upper left on display
lcd_putsf("%: "); //string from flash
lcd_gotoxy(0,3); //position to upper left on display
lcd_putsf(" "); //string from flash

flag=0;
//convert to percentage
for(i=0; i<5; i++)
begin
if(abs(percent[i])>99)
flag=1;
end
if(flag == 0)
begin
lcd_gotoxy(2,2); //position to upper left on display
per1=abs(percent[0]);
per2=abs(percent[1]);
per3=abs(percent[2]);
per4=abs(percent[3]);
per5=abs(percent[4]);
sprintf(lcd_buffer,"%2d %2d %2d %2d %2d", per1,per2,per3,per4,per5);
lcd_puts(lcd_buffer);
end
else
begin
lcd_gotoxy(2,2); //position to upper left on display
lcd_putsf("-- -- -- -- --");
end
lcd_gotoxy(0,3);
lcd_putsf(" ");
end // else

end //else if (~PIND.7)

end

//**********************************************************
//Task 2 print the system time
void task2(void)
begin
time2=t2; //reset the task timer
puts_int();
end

//**********************************************************
//Task 3 print the system time
void task3(void)
begin

time3=t3; //reset the task timer
switch (PushState)
begin
case NoPush:
if (PIND.6 != 0) PushState=MaybePush;
else PushState=NoPush;
break;
case MaybePush:
if (PIND.6 != 0)
begin
PushState=Pushed;
PushFlag=1;
end
else PushState=NoPush;
break;
case Pushed:
if (PIND.6 != 0) PushState=Pushed;
else PushState=MaybeNoPush;
break;
case MaybeNoPush:
if (PIND.6 != 0) PushState=Pushed;
else
begin
PushState=NoPush;
PushFlag=0;
end
break;
end
end

void task4(void)
begin
time4 = t4;
while(time4>0);
end
//**********************************************************
// -- non-blocking keyboard check initializes ISR-driven
// receive. This routine merely sets up the ISR, which then
//does all the work of getting a command.
void gets_int(void)
begin
r_ready=0;
r_index=0;
UCSRB.7=1;
end

//**********************************************************
// -- nonblocking print: initializes ISR-driven
// transmit. This routine merely sets up the ISR, then
//send one character, The ISR does all the work.
void puts_int(void)
begin
t_ready=0;
t_index=0;
if (t_buffer[0]>0)
begin
putchar(t_buffer[0]);
UCSRB.5=1;
end
end

void calc_percent(int my_letter, int pos[5])
begin
finger f;
int alphaf[5], posf[5];
for (f=thumb;f<=pinky;f++)
begin
alphaf[f] = alpha[my_letter][f];
posf[f] = pos[f];
percent[f]=((100*(alphaf[f]-posf[f]))/(alphaf[f]));
end
end

void plusminus()
begin
//negative percentages means bend fingers
finger f;
for (f=thumb;f<=pinky;f++)
begin
if(percent[f]<0)
begin
dir[f]='-';
end
else
begin
dir[f]='+';
end
end
end

//matches finger positions to appropriate letter
int match(int pos[])
begin
int end_index = 26, i,j=0;
int alpha_index[26];
finger f;
bool found=FALSE;

for(i=0;i<26;i++) alpha_index[i]=i;

for (f=thumb;f<=pinky;f++)
begin
found = FALSE;
for(i=0;i<end_index;i++)
begin
//found finger position
if ((alpha[alpha_index[i]][f] >= pos[f]-RANGE) && (alpha[alpha_index[i]][f] <= pos[f]+RANGE))
begin
alpha_index[j++]=alpha_index[i];
found = TRUE;
end
end

//if no position is found for the list
//of possible letters, then exit loop
if (found==FALSE) return NOTFOUND;

end_index = j;
j=0;
end

return alpha_index[0];
end

void set_values(letter l, int arr[])
begin

finger f;
for (f=thumb;f<=pinky;f++)
begin
alpha[l][f]=arr[f];
end

index_set[N++] = (int) l;
end

void main(void)
begin

lcd_init(LCDwidth); //initialize the display
lcd_clear();
//set up timer 0

//init vars and ports
PushState = NoPush;
PushFlag = 0;
DDRB = 0xff;
PORTB = 0x00;
DDRD=0x3f; // PORT A is an output
PORTD = 0xff;

//init the A to D converter
//channel zero/ left adj /EXTERNAL Aref
//!!!CONNECT Aref jumper!!!!
ADMUX = 0x60;
//enable ADC and set prescaler to 1/128*16MHz=125,000
//and clear interupt enable
//and start a conversion
ADCSR = 0b11000111;

//serial setop for debugging using printf, etc.
UCSRB = 0x18;
UBRRL = 103;
putsf("\r\nStarting...\r\n");

//set up timer 0
OCR0=250; //1 mSec
TIMSK=2; //turn on timer 0 cmp-match ISR
TCCR0=0b00001011; //prescalar to 64 and Clr-on-match

//init the task timers
time1=t1;
time2=t2;
time3=t3;

r_ready=0;
t_ready=1;

//crank up the ISRs
#asm
sei
#endasm

gets_int();
lcd_clear();
lcd_gotoxy(2,1);
lcd_putsf("SIGN LANGUAGE");

lcd_gotoxy(3,2);
lcd_putsf("INTERPRETER");
task4();
task4();

printf("BEGIN HERE......");

// measure and display loop
while (1)
begin
if (time1==0) task1();
if (time2==0) task2();
if (time3==0) task3();

end
end