Changeset 30c5aab for perl/lib/BarnOwl/Parse.pm
- Timestamp:
- Mar 7, 2010, 6:54:59 PM (14 years ago)
- Branches:
- master, release-1.10, release-1.6, release-1.7, release-1.8, release-1.9
- Children:
- d64f963
- Parents:
- 2d232ed
- git-author:
- David Benjamin <davidben@mit.edu> (02/27/10 22:16:24)
- git-committer:
- David Benjamin <davidben@mit.edu> (03/07/10 18:54:59)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
perl/lib/BarnOwl/Parse.pm
r2d232ed r30c5aab 5 5 6 6 use base qw(Exporter); 7 our @EXPORT_OK = qw(tokenize );7 our @EXPORT_OK = qw(tokenize tokenize_with_point); 8 8 9 9 # TODO: have the main function return whether or not it was a valid parse, with … … 33 33 my $space = qr{[ \t]}; 34 34 35 sub tokenize {35 sub tokenize_with_point { 36 36 my $line = shift; 37 37 my $point = shift; … … 94 94 return (\@words, $cword, $word_point, $cword_start, $cword_end); 95 95 } 96 97 sub tokenize { 98 my $line = shift; 99 100 my ($words, $word, $word_point, 101 $word_start, $word_end) = tokenize_with_point($line, 0); 102 return $words; 103 }
Note: See TracChangeset
for help on using the changeset viewer.