Content Extraction and Transmission v. San Diego County Credit Union

1
CET v. SDCCU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Zimmerman & Weiser LLP
Anatoly S. Weiser
3525 Del Mar Heights Road, #295
San Diego, CA 92130
Tel: (858) 720-9431
Fax: (858) 947-3950
aw@ iplcounsel.com

Jean-Marc Zimmerman (Pro Hac Vice Pending)
226 St. Paul Street
Westfield, NJ 07090
Tel: (908) 654-8000
Fax: (908) 654-7207
jmz@iplcounsel.com

Attorneys for plaintiff Content Extraction and Transmission LLC

CONTENT EXTRACTION AND
TRANSMISSION LLC

Plaintiff,

v.

SAN DIEGO COUNTY CREDIT UNION,

Defendant.

Case No.

COMPLAINT FOR PATENT
INFRINGEMENT
DEMAND FOR JURY TRIAL

Plaintiff Content Extraction and Transmission LLC (CET) demands a jury trial and
complains against defendant San Diego County Credit Union (defendant), as follows:
THE PARTIES
1. CET is a limited liability company organized and existing under the laws of the
State of New Jersey, with its principal place of business at New Jersey.
2. Upon information and belief, defendant is a company with a principal place of
business located in San Diego, California.
3. Defendant has bank branches throughout this judicial district in which it is
conducting business.

'12CV1878 RBB BTM
Case 3:12-cv-01878-BTM-RBB Document 1 Filed 07/30/12 Page 1 of 66

2
CET v. SDCCU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
JURISDICTION AND VENUE
4. This action arises under the patent laws of the United States of America, Title 35 of
the United States Code. This Court has jurisdiction of this action under 28 U.S.C. 1331 and
1338(a).
5. Defendant is doing business and committing infringements in this judicial district
and is subject to personal jurisdiction in this judicial district.
6. Venue is proper in this judicial district pursuant to 28 U.S.C. 1391 and 1400(b).
CLAIM FOR PATENT INFRINGEMENT
7. CET repeats and incorporates herein the entirety of the allegations contained in
paragraphs 1 through 6 above.
8. On November 2, 1993, U.S. Patent No. 5,258,855 (hereinafter referred to as the
855 patent) was duly and legally issued for an invention entitled Information Processing
Methodology.
9. On November 29, 1994, U.S. Patent No. 5,369,508 (hereinafter referred to as the
508 patent) was duly and legally issued for an invention entitled Information Processing
Methodology.
10. On April 29, 1997, U.S. Patent No. 5,625,465 (hereinafter referred to as the 465
patent) was duly and legally issued for an invention entitled Information Processing
Methodology. A copy of the 465 patent is attached to this Complaint as Exhibit 1.
11. On June 16, 1998, U.S. Patent No. 5,768,416 (hereinafter referred to as the 416
Methodology. A copy of the 416 patent is attached to this Complaint as Exhibit 2.
12. On August 21, 2007, U.S. Patent No. 7,259,887 (hereinafter referred to as the 887
Methodology.

3
CET v. SDCCU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
13. On January 6, 2009, U.S. Patent No. 7,474,434 (hereinafter referred to as the 434
Methodology.
14. CET is the owner by way of assignment of all right, title and interest in and to the
855, 508, 465, 416, 887 and 434 patents. The 465 and 416 patents will hereinafter be
collectively referred to as the Patents-in-Suit.
COUNT ONE
15. CET repeats and incorporates herein the entirety of the allegations contained in
paragraphs 1 through 6 and 8 through 14 above.
16. Defendant has and still is infringing, actively inducing the infringement of and
contributorily infringing in this judicial district the 465 patent by processing check and cash
deposits made by customers at its Automatic Teller Machines (ATMs) using its envelope-free
deposit service, and by processing check deposits made from mobile electronic devices using its
SDCCU Mobile Deposit service.
17. Defendant's envelope-free deposit service extracts information from checks and cash
deposited at its ATMs and then transmits such extracted information to an application program to
process the deposits, in a manner defined by the claims of the 465 patent, without permission from
CET.
18. Defendant's SDCCU Mobile Deposit service extracts information from images of
checks taken using a mobile electronic device and then transmits such extracted information to an
application program to process the deposit, in a manner defined by the claims of the 465 patent,
without permission from CET.
19. CET has been damaged by such activities of the defendant which infringe the 465
patent and will be irreparably harmed unless such infringing activities are enjoined by this Court.

4
CET v. SDCCU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
COUNT TWO
20. Plaintiff repeats and incorporates herein the entirety of the allegations contained in
paragraphs 1 through 6, 8 through 14, and 16 through 19 above.
21. Defendant has and still is infringing, actively inducing the infringement of and
contributorily infringing in this judicial district the 416 patent by processing check and cash
deposits made by customers at its ATMs using its envelope-free deposit service, and by processing
check deposits made from mobile electronic devices using its SDCCU Mobile Deposit service.
22. Defendant's envelope-free deposit service extracts information from checks and cash
deposited at its ATMs and then transmits such extracted information to an application program to
process the deposits, in a manner defined by the claims of the 416 patent, without permission from
CET.
23. Defendant's SDCCU Mobile Deposit service extracts information from images of
checks taken using a mobile electronic device and then transmits such extracted information to an
application program to process the deposit, in a manner defined by the claims of the 416 patent,
without permission from CET.
24. CET has been damaged by such activities of the defendant which infringe the 416
patent and will be irreparably harmed unless such infringing activities are enjoined by this Court.
PRAYER FOR RELIEF
WHEREFORE, plaintiff CET prays for judgment against the defendant on all the counts and
for the following relief:
A. Declaration that the CET is the owner of the Patents-in-Suit and has the right to sue
and to recover for infringement thereof;
B. Declaration that the defendant has infringed, actively induced the infringement of,
and contributorily infringed the Patents-in-Suit;
C. A preliminary and permanent injunction against the defendant, each of its officers,
agents, servants, employees, and attorneys, all parent and subsidiary corporations,

5
CET v. SDCCU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
their assigns and successors in interest, and those persons acting in active concert or
participation with them, including distributors and customers, enjoining them from
continuing acts of infringement, active inducement of infringement, and contributory
infringement of CETs 465 and 416 patents;
D. An accounting for damages under 35 U.S.C. 284 for infringement of CETs 465
and 416 patents by the defendant and the award of damages so ascertained to the
CET together with interest as provided by law;
E. Award of CETs costs and expenses; and
F. Such other and further relief as this Court may deem proper, just and equitable.
DEMAND FOR JURY TRIAL
Plaintiff CET demands a trial by jury of all issues properly triable by jury in this action.

Dated: July 27, 2012 Zimmerman & Weiser LLP

By: /s/ Anatoly S. Weiser
Anatoly S. Weiser
3525 Del Mar Heights Road, #295
San Diego, CA 92130
Tel: (858) 720-9431
Fax: (858) 947-3950
aw@ iplcounsel.com
Attorneys for plaintiff Content Extraction and Transmission LLC

6

EXHIBIT TABLE OF CONTENTS

Exhibit 1 p 7

Exhibit 2 p 38

111111 1111111111111111111111111111111111111111111111111111111111111
United States Patent [19]
Lech et al.
[54] INFORMATION PROCESSING
METHODOLOGY
[75] Inventors: Robert Lech, Jackson; Mitchell A.
Medina, Essex Fells; Catherine B.
Elias, Plainsboro, all of N.J.
[73] Assignee: fiternational Patent Holdings Ltd.,
Hamilton, Bermuda
[21] Appl. No.: 348,224
[22] Filed: Nov. 28, 1994
[63]
[51]
[52]
[58]
[56]
Related U.S. Application Data
Continuation of Ser. No. 143,135, Oct. 29, 1993, Pat. No.
5,369,508, which is a continuation of Ser. No. 672,865, Mar.
20, 1991, Pat. No. 5,258,855.
fit. CI.
6
....................................................... H04N 1/40
U.S. CI. .......................... 358/448; 358/403; 358/406;
382/209; 382/175; 382/180; 395/768
Field of Search ..................................... 382/209, 175,
4,034,343
4,667,248
4,760,606
5,034,990
5,095,445
5,140,650
5,148,520
382/180,317; 358/462. 448, 400, 401,
403,447,449,451,452.453.460,463,
467.470,471, 474
References Cited
U.S. PiITENT DOCUMENTS
7/1977 Wllmer ........................ 340/146.3 MA
5/1987 Kanno ..................................... 358/280
7/1988 Lesnick et aI. .. ......................... 382148
7/1991 Klees ........................................ 382122
3/1992 Sekiguchi ................................ 364/514
8/1992 Casey et aI ............................... 382/61
9/1992 Morgan ................................... 395/148
US005625465A
[11] Patent Number:
[45] Date of Patent:
5,625,465
Apr. 29, 1997
5,153,927 10/1992 Yamanari .................................. 382/61
5,228,100 7/1993 Takeda et aI ........................... 382/175
5,258,855 1111993 Lech et aI ............................... 358/462
5,367,619 1111994 Dipaolo et aI .......................... 395/149
OTHER PUBLICATIONS
"Kurzweil 5200 Intelligent Scanning System", Xerox Imag-
ing Systems. Inc .. 1990.
Que's Computer User's Dictionary, 2nd Ed . Bryan Pfaffen-
berger (author); 1991; p. 144.
Primary Examiner-Joseph Mancuso
Assistant Examiner-Jerome Grant. II
Attorney, Agent, or Firm-Foley & Lardner
[57] ABSTRACT
An information processing methodology gives rise to an
application program interface which includes an automated
digitizing unit, such as a scanner. which inputs information
from a diversity of hard copy documents and stores infor-
mation from the hard copy documents into a memory as
stored document information. Portions of the stored docu-
ment information are selected in accordance with content
instructions which designate portions of the stored document
information required by a particular application program.
The selected stored document information is then placed
into the transmission format required by a particular appli-
cation program in accordance with transmission format
instructions. After the information has been transmission
formatted, the information is transmitted to the application
program. In one operational mode, the interface interactively
prompts the user to identify, on a display, portions of the
hard copy documents containing information used in appli-
cation programs or for storage.
53 Claims, 11 Drawing Sheets
~ ____ ---, 10
I XYZ Corporation ~
Customer
ABC Corporation
123 Sixteenth Street
Hometown,NJ 08901
~ O
JO
c
lO
10
I Account Number 1123456789 ho
,r
10 10 10
1-'"
10
10
10
, .....
\ ~
10
10
r'
Mail To:
XYZ Corporation
PO Box 567
Anywhere, NY 63130
~
10
Previous Balance
New Charges
Other Debits
Finance Charges(lO%)
Payments
Other Credits
New Balance
~
10
Winter Sale In Effect Throughout January ~
30)
10
r'
$1234.56
$789.01
$2.34
$1000.00
$1025.91
~ O
~
~
~
o
o
~ .
o
~
Exhibit 1
7
u.s. Patent Apr. 29, 1997 Sheet 1 of 11 5,625,465
z
cf
6"
s
C
Ir
C
>. 1 .Q
a 0
LO
"-
to
"-
a
-
l"-
.,.:
cO
Q.)
o- N o-
N
'-E
N
-
I"- u- u-
.!:
C\J
._ c

._ c
Oct)
L-
a.::> a.::>
EE 0-
c.. 0-
c:{ <:(
0
j
<:;t
,
N,
I
C\l
a
t()
<:;t
a ; ~ ~
'}J
~ ! ' C )
_C\J
r
C\l
:::J
CI..
E
8L-
to
t()
C\l
>.
L-
a
0
E
C\l
~
C\l
Q.)
1/
E
0
0
C\J
'-
0
Q.)
C\J
c
,/
c
0
u
(f)
a
Exhibit 1
8
u.s. Patent Apr. 29, 1997 Sheet 2 of 11
.--------, 10
I XYZ Corporation ~
Customer 10
~ - - - - - - - - - - - - - - - - ~ ~
ABC Corporation
123 Six.teenth Street
Hometown, NJ 08901
10
f-.J
c
lO
10
I Account Number 1123456789 P
IO
./"
/" 10 10 10
r 1/
Statement Date: 12/01/86 \ ....
10
10
10 10
Payment Date: 1101187
10../
,/
\/"
\ .......
/
10
Mail To:
XYZ Corporation
PO Box 567
Anywhere, NY 63130
'--10
Previous Balance
New Charges
Other Debits
Finance Charges(lO%)
Payments
Other Credits
New Balance
\".
10
Winter Sale In Effect Throughout January ~
)
30/
FIG. 2
5,625,465
10
L
$1234.56
$789.01
$2.34
$1000.00
$1025.91
I
I-'
o
o
o
I
~
I
~
I o
o ~
I--'
I o
~
Exhibit 1
9
5,625,465
Sheet 3 of 11
Apr. 29, 1997
u.s. Patent
~
0
N
c
X "-
~
-
l.L
Exhibit 1
10
@
/232A
I XYZ CORPORATION I
CUSTOMER MAIL TO:
ABC CORPORATION XYZ CORPORATION
123 SIXTEENTH STREET PO BOX 567
HOMETOWN, NJ 08901 ANYWHERE, NY 63130
IACCOUNT NO. 11234567891 PREVIOUS
BALANCE
$1234.56
[STATEMENT DATE 112/01/87 I NEW
$789.01
I PAYMENT DATE 1 1/01/87 1 CHARGES
OTHER
DEBITS
FINANCE
$2.34
CHARGES 10%
PAYMENTS $1000.00
WINTER SALE IN EFFECT
OTHER
THROUGHOUTJANUARY!
CREDITS
NEW
BALANCE
$1825.91
~
232L
FIG. 38
L
~
/
ACCOUNTS PAYABLE
,232C
VENDOR c:::J
INVOICE #
INVOICE DATE
DATE DUE
AMOUNT DUE
232R--"
~
~
"\
\
230.-'
~

rJ1

~
=
~
~
a
>
"0
rt
N
~
I-"
~
-...J
ga
~
II
""'
S,
I-"
I-"
OJ
-...
Q\
N
OJ
-...
~
Q\
OJ
Exhibit 1
11
u.s. Patent
Apr. 29, 1997
Sheet 5 of 11
5,625,465
User
Dialog
(1.0
2.0
3.0)
Input Output
Input
Document
Info
Document
Output
Process Process Process

Transmission
4.0
~
Comm.
External
Device
FIG. 4
Exhibit 1
12
u.s. Patent Apr. 29, 1997
(/)
(/)
0Q)
4- U
C:O
-\...,
0....
c
-Q)
:J E
0.:::1
c: U
0
0
>-
xc. \...
~ 8
'\
-
rn
-
-
~ -
0:::
c:J
.J::.o.
U
uE
Sheet 6 of 11
0
(
Q)
0\ >-
00.
EO
-u
(\j
-
d)_
0l:J
00.
Ec
--
0
-
\...,
C\J
Q)
c:
c:
0
U
(J)
o
~
E
E
o
u
E
o
\...,
l1..
5,625,465
-
Exhibit 1
13
Input
Process
1.0
Input
Document
Document
1. .2.2
Mntn
Def 1St-----,
Dialog
User
Dialog
2.1
Mntn
Library
Output
Document
l
Document 2.3
, ) )
Definitions
Dialog
,r-
Process
Doc
Transmission ~
'- -'User l .. ,/ Comm.
Dialog
Diolog(1 J
FIG. 6
External
Device
2.0
~
4.0
Output
Process
3.0
J
~

rJ'J.

~
=
~
~
a
>
't:I
:;
N
~
t-<
\0
\0
-...J
~
('C)
II
-...J
~
t-<
t-<
OJ
-- ~
N
OJ
~
~
OJ
Exhibit 1
14
1.0
Input
Process
Document
File
3.0
Input
Document
2.1.1
Document
2.1.3,
Document, Output
Errors
2.1.2

Store
Doc
Correct
Errors
FIG. 7
h
Dialog Dialog

Dialog
'- User
I-"
,-
Retr
Doc
Document
Output
Process
Document
Document
,

Mntn
Def's
..,
f--'2.1
2.3
t )
Process
Doc

\Jl

f""!'-
a
>

N

I-"

-....l

00
S,
I-"
I-"
Ul

0\
N
Of

0\
Ul
Exhibit 1
15
2.1
2.2.1)
Input
Mntn
Document
Define
Library
Tempi
Document
Selective
Variable
Template
Scan
Definition
Instructions
Dialog
I
Content
2.2.2
Data
Format
)
Relationship
Instr.
Define
Definition
User
Dialog
Rei
Transmission ~ Dialog
Format
Physical
Instructions
223)
Format
Define
Definition
Format
- --_ .. -
FIG. 8
2.24 Definitions
\...
}
r--
Select
Def's
Dialog
Definitions
21
Definition
File
1---
2
.
2
~
2.3,
Process
Doc
)
User
o

rJ1

~
=
f""I'.
a
~
:-;
N
,,'C
I-'
'C
'C
~
ga
~
a
'C
~
I-'
I-'
OJ
~
="
N
OJ
~
~
="
OJ
Exhibit 1
16
2.1
-,
2.2

Mntn
Library
Mntn
Def's
Document
Selective
Scan
Information

Content
Format
Information

/
Transmission
Format
Information

Extract
Data
Dialog
Document
Data
2.3.2
1 t

Pre AppJ
Process
Dialog
2.3.3
'---
FIG. 9
User
To Bus 240
) I (
Input
File
Comm U.O
2.3

o

00

=
"'"'""
a

N
",\CI
......
\CI
\CI
-...J

I'D
::a.
......
c
S,
......
......
01
'-II
0\

01
'-II

0\
01
Exhibit 1
17
u.s. Patent
Apr. 29, 1997
Sheet 11 of 11
5,625,465
-
00,)
Cu
~ . -
0,
-<U
><0
W
0
\...
E
E
-
0
u
0
r0i
-
-
N
a>"S
r<)
,-::J
!'Ii,-
"-
00.
Olo.
~ -
0-
U::J E::J
0 -0
o
-
.
CD
-
~
-0.
Q)
Xo C ' I ~
~ u
00.
EO
_u
-
_c
::J<U
o.E
-::J
::J
u
00
0\
If)
f\O
r.n
0(1)
-u
.so
'-
.
a...
(\j
Exhibit 1
18
5,625,465
1
INFORMATION PROCESSING
METHODOLOGY
2
by a business increases relative to the number of forms that
must be processed, it becomes less cost-effective to design
a dedicated machine for processing each type of form
format. This problem is particularly significant in small
This application is a continuation of application Ser. No.
081143.135, filed Oct. 29, 1993, now U.S. Pat. No. 5369,
508 which is a continuation of application Ser. No. 07/672,
865, filed Mar. 20, 1991, now U.S. Pat. No. 5,258,855,
issued Nov. 2, 1993.
5 businesses which may, for example, receive fifty invoices a
month, all in different, non-fixed, formats. It is frequently
not cost-effective for a small business to design dedicated
systems for inputting information in each of these various
formats. This leaves a small business with no other practical
BACKGROUND OF THE lNVENTION
10 alternative than to manually input the information off of each
invoice each month.
The invention is directed to a system for efficiently
processing information originating from hard copy docu-
ments. More specifically, the invention is directed to a hard
copy document to application program interface which
minimizes the need to manually process hard copy docu- 15
ments.
In the past, information contained on hard copy docu-
ments was manually entered into a computer via the input
controller of a particular computer. The original document
was then filed away for future reference. Automatic input of 20
data was limited to the input of Magnetic Ink Character
Recognition (MICR) data and to Optical Character Recog-
nition (OCR) data. This fixed-position data was forwarded
directly to a dedicated computer application specifically
designed to accommodate the input format In more recent 25
years, typewritten text has been mechanically inputted into
a computer via a text file. Examples of this latter type of
system are word processors and photo-typesetters.
These conventional systems have limitations which 30
decrease the efficiency of processing information from a
hard copy document. For example, the systems discussed
above are limited in their application to MICR, OCR, or
typewritten data. Parsing and processing data is limited to
the particular requirements of the particular computer appli- 35
cation which requires the input data. In addition, in these
conventional systems, the actual hard copy document must
be retained for future reference at great expense.
In a sophisticated computer network, different users may
require different portions of the information contained on a 40
hard copy document. For example, if the hard copy docu-
ment is an invoice returned with payment of a bill, the
accounting department may need all of the monetary infor-
mation contained on the bill while the mailroom may need
only customer address information, to update a customer's 45
address. Therefore, there is a need for a system in which
specific information from a hard copy document can be
selectively distributed to various users.
Another problem with conventional systems is that users,
even within the same company, may require that the infor- 50
mation extracted from a hard copy document be transmitted
to a particular application program in a specific transmission
format. For example, one department in a company may use
a particular application program which must receive infor-
mation using a particular character as a delimiter and other 55
departments may require the information in a different
format using different delimiters.
Another problem. particularly for small businesses, is that
current systems can not efficiently accommodate the input-
ting of information from a diversity of hard copy documents. 60
A large business which receives many forms in the same
format can afford a system which inputs a high volume of
information in that format into memory. For example, it is
cost-effective for a bank which processes hundreds of thou-
sands of checks a month to buy a dedicated machine which 65
can read information off of checks having a rigidly defined,
or fixed, format However, as the diversity of forms received
SUMMARY OF THE lNVENTION
It is an object of the invention, therefore, to provide an
application program interface which allows a user to select
specific portions of information extracted from a diversity of
hard copy documents and allows the user to direct portions
of this information to several different users in accordance
with the needs of the particular user.
It is also an object of the invention to provide a cost-
effective system for inputting hard copy documents which
can accommodate hard copy documents in a diversity of
formats.
It is another object of the invention to provide an appli-
cation program interface which allows a user to put
information, which is to be transmitted, into a particular
transmission format. based upon the needs of the receiver of
the information.
It is a further object of the invention to provide an
application program interface which will allow the
extraction, selection, formatting, routing, and storage of
information from a hard copy document in a comprehensive
manner such that the hard copy document itself need not be
retained.
It is another object of the invention to provide a system
which reduces the amount of manual labor required to
process information originating from a hard copy document.
A further object of the invention is to reduce the time
required to process information originating from a hard copy
document so that a higher volume of transactions involving
hard copy documents can be processed.
The invention provides an application program interface
which inputs a diversity of hard copy documents using an
automated digitizing unit and which stores information from
the hard copy documents in a memory as stored document
information. Portions of the stored document information
are selected in accordance with content instructions which
define portions of the stored document information required
by a particular application unit. Selected stored document
information is then formatted into the transmission format
used by the particular application program based on trans-
mission format instructions. The transmission formatted
selected stored document information is then transmitted to
the particular application program. The hard copy docu-
ments may contain textual information or image information
or both.
The interface operates in three different modes.
In a first mode, the interface extracts all of the information
from hard copy documents and stores this information in
memory. Parsing of various portions of the extracted infor-
mation is performed in accordance with content instructions.
In a second mode, the user operates interactively with the
interface by use of a display and an input device, such as a
mouse. In this second mode, a hard copy document is
inputted and displayed on the display. The interface then
Exhibit 1
19
5,625,465
3 4
prompts the user to identify the location of various infor-
mation. For example. the interface can ask the user to
identify the location of address information on the hard copy
document. In response. the user positions the mouse to
identify address information using a cursor. The identified 5
information is then stored as address information in memory.
Subsequently. the interface again prompts the user to iden-
tify other pieces of information. which are then stored in the
appropriate locations in memory. This process proceeds until
Appendix 14 illustrates another example of a hard copy
document containing information to be processed by the
instant invention.
DETAilED DESCRIPTION OF THE
PREFERRED EMBODIMENTS
Hardware
The invention provides an interface between information
originating from a hard copy document and a computer
application unit which uses the information. The computer
application unit can be a particular computer application
program or a device which is controlled in accordance with
instructions or information from the hard copy document.
all of the information which is desired to be extracted off of 10
the hard copy document is stored in memory.
In a third mode of operation. selected portions of infor-
mation are extracted off of hard copy documents in accor-
dance with predetermined location information which has
been specified by the user. For example. the user can define
a template which specifies the location of information on
hard copy documents. Templates can be formed in conjunc-
tion with second mode operation. Alternatively. the user can
instruct the interface to search hard copy documents for a
particular character or symbol. located on the hard copy
documents. The information desired to be extracted off of
the hard copy documents is specified relative to the location
of this character or symbol.
The interface can also prompt or receive from an appli-
cations program or another information processing system.
required information. content instructions. and format
instructions.
Other objects. features. and advantages of the invention
will be apparent from the following detailed description of
the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention will be described in further detail below
with reference to the accompanying drawings. in which:
FIG. 1 illustrates hardware for implementing a preferred
embodiment of the instant invention;
FIG. 2 illustrates an example of a hard copy document
containing information to be processed by the instant inven-
tion;
FIGS. 3A and 3B are enlarged views of the computer of
FIG. 1 used to explain how the invention interactively
prompts a user to identify information;
FIG. 4 is an overall data flow diagram for the FIG. 1
preferred embodiment;
FIG. 5 is a detailed input data flow diagram for the FIG.
1 preferred embodiment;
FIG. 6 is a detailed information processing data flow
diagram for the FIG. 1 preferred embodiment;
15 The invention also allows storing a copy of the hard copy
document in a memory and retrieving the copy of the hard
copy document. By providing a comprehensive and inte-
grated system which can accommodate almost all of the
possible uses of information contained on a hard copy
20 document. the instant invention allows for a paperless office.
The invention includes hardware and software necessary
to extract. retrieve. and process information from the hard
copy document. A copy of the actual image of the hard copy
docuI)1ent is stored in memory. Textual information
25 extracted from the hard copy document is also stored in
memory. Textual information is information. such as alpha-
numeric characters. which is recognized on the hard copy
document and which is stored in a form which corresponds
to the particular recognized character. For example. the
30 extracted characters can be stored in the ASCn format in an
electronic memory.
The user can have all of the information extracted from
the hard copy document and stored in memory.
Alternatively. the interface can interactively prompt the user
35 to identify specific pieces of information for storage. The
interface can also extract specific pieces of information
using a predefined template. The interface can also prompt
or receive from another information processing system or an
applications program desired information. content
40 instructions. and format instructions.
45
The instant invention also provides for parsing informa-
tion extracted from the hard copy document and for direct-
ing this parsed information to specific users or application
programs as an input file.
The invention also permits the user to define the trans-
mission format of the input file for a particular computer
application unit.
FIG. 7 is a more detailed information processing data flow 50
diagram for the maintain library module of FIG. 6;
FIG. 1 illustrates hardware for implementing a preferred
embodiment of a hard copy document to application pro-
gram interface according to the instant invention. The inter-
face 200 processes information extracted off of hard copy
document 100 and provides information to application units
270 in a form required by each particular application unit.
The interface extracts information off of a hard copy docu-
ment 100 utilizing a scanner 210. The scanner 210 can be
any type of scanner which extracts information off of hard
copy documents. for example, an Optical Reader.
FIG. 8 is a more detailed information processing data flow
diagram for the maintain definitions module of FIG. 6;
FIG. 9 is a more detailed information processing data flow 55
diagram for the process document module of FIG. 6;
FIG. 10 is a detailed output data flow diagram for the FIG.
The scanned information is stored in a scanner memory Appendix 11 lists data corresponding to the hard copy
document of FIG. 2;
Appendices 12A. 12B. and 12C illustrate examples of
data which can be selected from the extracted data of FIG.
11 in accordance with content instructions.
60 220 or in main memory 250. as will be described in greater
detail below. If main memory 250 or another memory is
available to store the scanned information. then scanner
memory 220 can be omitted.
Appendices 13A. 13B. and 13C illustrate examples of the
data of FIGS. 12A. 12B. and 12C formatted in accordance 65
with various transmission format instructions to form input
files; and
The information from scanner memory 220 or main
memory 250 is transmitted to computer 230. In the preferred
embodiment. computer 230 includes a display 232. a key-
board 234. and a mouse 236. The display 232 displays an
Exhibit 1
20
5,625,465
5
image of the hard copy document itself and/or information
necessary to process the information extracted off of the hard
copy document.
The computer 230 is used to select portions of the stored
document information contained in memory in accordance 5
with content instructions which define portions of the stored
document information required by an application unit. These
content instructions may be provided by the application
program. Alternatively, the content instructions can be input-
ted via an input device such as a keyboard, a touch screen, 10
a mouse, a notepad, a voice recognition device, or the like.
The computer 230 is also used to format selected stored
document information into the transmission format used by
an application unit based on transmission format instruc-
tions. The transmission format instructions may be provided 15
by the application program. Alternatively, the transmission
format instructions can be inputted via a keyboard, a touch
screen. a mouse, a notepad, a voice recognition device, or
the like.
6
ASCII format. The scanner memory 220 can be, for
example, an electronic, magnetic, or optical memory.
FIG. 3A illustrates an enlarged view of the computer 230
of FIG. 1. This view will be used to describe a second mode
of operation. In this second mode of operation, the hard copy
document 100 is scanned and a copy of the document 100 is
displayed on display 232 of computer 230, based on the
contents of information temporarily stored in scanner
memory 220. After the document is displayed on display
232, the computer 230 interactively prompts the user to
identify the location of specific pieces of information on the
hard copy document. In the FIG. 3A illustration, this prompt
message is indicated as the message beginning with the
arrow.
For example, the prompt message can ask the user to
Thus, the computer 230 is used to generate an input file
for a particular application unit. The computer 230 is con-
nected to scanner memory 220, main, or permanent,
memory 250. a printer 260, and application units 270, via a
bus 240. Although FIG. 1 illustrates use of a bus to connect
components together, it is understood that any routing or
connecting link, implemented in hardware or software or
both. can be employed instead of, or in addition to, a bus.
Instructions to or in the computer 230 control the main
memory 250, the printer 260, the application units 270, and
the bus 240. Instructions to or in computer 230 can also
control exchanges of information with scanner memory 220.
identify the location of account number information on the
hard copy document. The user then uses an input device,
such as keyboard 234 or mouse 236 or a touch screen,
notepad, voice recognition device, or other input device to
20 position a cursor on the display to identify the location of the
information requested by the prompt message. For example,
the cursor could be used to define a block (which could be
highlighted) containing the requested information, followed
by a mouse "enter" click. In this example, the user would
25 move the mouse to identify the location of the account
number information contained on the hard copy document
100. The computer 230 then stores the information which
has been identified by the user as account number informa-
tion in the appropriate address or subfile or as the appropri-
When the computer 230 generates an input file for a
particular document, the computer 230 can send this input
file directly to an application unit or can store this input file
30 ate variable or parameter, or data field in memory. The
computer then prompts the user to identify the location of
other information on the hard copy document, such as,
statement date information. The process proceeds until all of
the desired information has been stored into the appropriate
35 locations in memory.
in the main memory 250 until required by an application
unit. The main memory 250 may also optionally store a copy
of the image information for the hard copy document and the
textual information for the hard copy document Thus, the
image information and textual information from the hard 40
copy document can be retrieved and printed out on printer
260. In addition, image and textual information stored in
scanner memory 220 or in main memory 250 can be used to
form additional input files at the time of input or at a later
time, based on content instructions and transmission format 45
instructions. Thus, the invention can, at the discretion of the
user, eliminate the need to retain copies of hard copy
documents, permitting a paperless office.
The application units 270 include particular application
programs and devices which are controlled in accordance 50
with information contained on hard copy document 100.
100 which contains information to be processed by the
instant invention. The document illustrated in FIG. 2 is a bill
from XYZ Corporation to customer ABC Corporation. FIG. 55
2 is only an example of a type of document that can be
processed by the instant invention.
In a first operational mode, the scanner 210 stores all of
the information extracted off of hard copy document 100 in
the scanner memory 220 or, alternatively, in main memory 60
250. The extracted information is stored in two forms. The
actual image of the hard copy document 100 is stored as
image information in the scanner memory 220. In addition,
the scanner memory 220 stores textual information recog-
nized on the hard copy document 100 by, for example, 65
employing standard character recognition software. In the
preferred embodiment, the textual information is stored in
FIG. 3B illustrates a variation of the second mode for
interactively prompting the user for information. In FIG. 3B,
the display is split into two portions. A left-hand portion
232L displays the image of the hard copy document and a
right-hand portion 232R displays the required application
program information. For example, in FIG. 3B, portion
232R displays a spreadsheet used by an application pro-
gram. While observing the split display, the user can input
instructions to associate specific pieces of information on the
hard copy document (for example, the vendor name indi-
cated by the mouse arrow 232A) with particular subfiles in
memory (for example, the vendor field next to which the
cursor 232C appears), using a mouse or other input device
(s) or both. The split display also allows the user to generate
content format instructions while observing the information
required for a particular application program on the right-
hand portion.
These second modes of operation are efficient for small
businesses which receive a small number of a wide variety
of invoices, since the user does not necessarily have to store
all of the information that appears on the hard copy docu-
ment. A further advantage is that data input is quicker, easier,
and more accurate than with previous keyboard methodol-
ogy. In addition, by specifying the location on the hard copy
document of information, the user may optionally create a
template, to be described in further detail below, for each
different type of invoice. This template is stored for future
use when another hard copy document in the same format is
received.
More specifically, instructions from computer 230 can
direct the scanner 210 and scanner memory 220, and/or main
Exhibit 1
21
5,625,465
7
memory 250. to scan and/or store only specific portions of
hard copy document 100. After the interactive prompts
required to obtain information for a desired application
program. the unused information stored in scanner memory
220 or 250 can be erased. Further. scanning of a second 5
identical document can be limited to only those portions of
the document which contain needed information.
8
The information processing module 2.0 is the module
which coordinates and drives the entire system. In the
preferred embodiment, the information processing module
2.0 is implemented primarily by computer 230.
FIG. 7 illustrates information processing data flow in the
maintain library module 2.1. The maintain library module
2.1 maintains a library of image information, for example, a
digitized image representing the actual appearance of the
hard copy document, and textual information of the hard
More specifically. in FIG. 2. the lines 10 drawn around
certain portions of the document represent the areas which
the user has previously identified as the portions of a
document to be extracted by the scanner 210 and stored in
scanner memory 220 and/or main memory 250. Since the
logo 20 and the message 30 have not been identified as an
area to be scanned and stored. these areas are not scanned
and stored in subsequent documents. Since the user has
previously associated each of the areas 10 with a specific
subfile of information. e.g .. the account number. the scanned
information is stored in memory locations corresponding to
that subfile.
10 copy documents for reference during processing. This
library can be incorporated within scanner memory 220,
main memory 250, or another independent memory, for
example, a RAM disk. The maintain library module 2.1
includes a store document module 2.1.1, a correct errors
15 module 2.1.2, a retrieve document module 2.1.3, and a
document file 2.1.4. These modules operate collectively to
store, retrieve, and correct document information.
Data Processing
FIGS. 4-10 illustrate the flow of data in the FIG. 1
preferred embodiment FIG. 4 illustrates the overall data
flow for the FIG. 1 preferred embodiment The preferred
embodiment includes an input process module 1.0. an infor-
mation processing module 2.0. and an output processing
module 3.0. The information processing module 2.0 is
equipped to receive instructions from and transmit informa-
tion to a user. The information processing module 2.0 can
also transmit to and receive information from a remote
external device through communication interface 4.0. Input
process module 1.0 and output processing module 3.0 can
also access communication interface 4. O. A module is imple-
mented in hardware. software. or a combination of hardware
and software. The specific implementation for a particular
business application depends upon a variety of factors. for
example. the relative costs of hardware and software imple-
mented systems. the frequency with which a user will want
The store document module 2.1.1, prior to routing the
document to the document file 2.1.4, may provide informa-
20 tion on recognition errors which may have occurred while
inputting the document For example, the store docuinent
module 2.1.1 identifies that a character contained on hard
copy document 100 was not recognized. The store document
module 2.1.1 also optionally causes a copy of the document
25 and its parsing to be displayed on the display 232 for
confirmation by the user. The user may utilize this oppor-
tunity to identify any errors in the displayed document and,
in conjunction with the correct errors module 2.1.2, to revise
the document's parsing, if necessary, prior to storage of the
30 document in memory. The module 2.1.1 also provides a
facility for the user to name a particular hard copy document
for cataloging, storage, and retrieval purposes. After the
document is named, the store document module 2.1.1 stores
to expand or modify the system. and the like.
FIG. 5 is a more detailed diagram of the input process
module 1.0 of FIG. 4. The input process module 1.0 includes
35
40
a character input module 1.1. an image input module 1.2,
and, in the preferred embodiment, a character recognition
device 1.3. The character input module inputs textual 45
information, such as alphanumeric characters, from an input
device such as keyboard 234. The image input module 1.2
inputs image information, for example, a digitized image of
the actual appearance of hard copy document 100. Textual
information can include textual input from an input device 50
such as keyboard 234 and textual information extracted from
the document by character recognition device 1.3. Both
types of information comprise an input document which is
transmitted to information processing module 2.0. In the
FIG. 1 preferred embodiment, the processing performed by 55
input process module 1.0 occurs in scanner memory 220,
computer 230, and main memory 250.
copies of the document in the document file 2.1.4.
The correct errors module 2.1.2 processes instructions
from the user to correct errors identified by the store
document module 2.1.1 and errors that have been spotted by
the user during the confirmation process.
The retrieve document module 2.1.3 permits the user to
retrieve a copy of a document previously stored in the
document file 2.1.4. As described above, long-term storage
is provided by main memory 250, if necessary.
FIG. 8 illustrates a more detailed information processing
data flow diagram for the maintain definitions module 2.2 of
FIG. 6. The maintain definitions module 2.2 allows the user
to define system and document parameters and maintains the
definitions of these system and document parameters. The
maintain definitions module 2.2 includes a define template
module 2.2.1 which allows the user to specify the location
of information on the document This information provided
by the user defines a template which is used to extract
information off the document and to associate the extracted
information with a particular variable or subfile. These
templates are illustrated by boxes 10 in the FIG. 2 example
of a hard copy document The maintain definitions module
2.2 can also access templates previously defined by the user
and stored in main memory 250. Templates can also be
provided as part of software packages developed by program
FIG. 6 illustrates information processing data flow for the
FIG. 1 preferred embodiment, that is, FIG. 6 illustrates data
flow in the information processing module 2.0. 60 developers.
The information processing module 2.0 includes a main-
tain library module 2.1, to be described in further detail
below in conjunction with FIG. 7, a maintain definitions
module 2.2, to be described in further detail below in
conjunction with FIG. 8, and a process document module 2.3 65
to be described in further detail below in conjunction with
FIG. 9.
The maintain definitions module 2.2 also includes a define
relationships module 2.2.2. The define relationships module
2.2.2 allows the user to define data relationships, or logical
relationships, between pieces of information extracted from
the hard copy document These pieces of information are
then used to generate an input file for a selected computer
application unit The user defines these relationships by
Exhibit 1
22
5,625,465
9 10
Operation
Examples of operation of a preferred embodiment will
now be described.
The user enters the system by providing instructions to the
content instructions. Alternatively, content instructions to
define relationships can be provided by application software.
If the user provides these content instructions. the content
instructions are inputted via keyboard 234 or via another
input device such as a notepad. a voice recognition device.
or the like. Examples of content instructions, data, and
logical relationships will be described in further detail in
conjunction with Appendices 11 and 12A. 12B. and 12C.
5 information processing module 2.0. The user then instructs
the information processing module 2 0 to conduct maintain
library processing, maintain definitions processing, or pro-
cess document processing.
If the user selects maintain library processing. the user The maintain definitions module 2.2 also includes a define
format module 2.2.3. The define format module 2.2.3 allows
the user to define transmission formats for an input file
which is then transmitted to a selected computer application
unit. Selection of the transmission format of the input file is
accomplished by the user through use of transmission format
instructions. Alternatively. the applications software itself
can generate its own transmission format instructions. When
the user must specify transmission format instructions. the
transmission format instructions are inputted via keyboard
234 or via another input device such as a notepad. a voice
recognition device. or the like. A further description of
various transmission formats will be provided below in
conjunction with Appendices 12A, 12B.12C.13A, 13B. and
13C.
10 then provides instructions to maintain or modify the docu-
ment library through the maintain library module 2.1. For
example. the user can direct the inputting and storage of a
hard copy document 100 or can retrieve and output a
document. The user requests inputting of a document
15 through the store document module 2.1.1. The system then
prompts the user to specify a storage location for the
inputted document. The document is then read in by the
input process module 1.0. A textual copy and/or an image
copy are stored into the document file 2.1.4. Errors which
have occurred during inputting are identified and corrected
20
A select definitions module 2.2.4 is also included in the
maintain definitions module The select definitions module 25
2.2.4 allows the user to store and select a set of definitions
to be used for processing the document. The definitions
identify pieces of information on the document by. for
example. absolute location. variable location, or relative
location. or by proximity to key words and/or symbols. 30
These definitions are described in further detail below by
way of an illustrative example.
data flow diagram for the process document module 2.3. The
35
process document module 2.3 processes the document after
the document has been stored in the system. The process
document module 2.3 gathers the appropriate information
which has been stored. and creates input file(s) 2.3.3 for the
selected application unit. The process document module 2.3
40
then transmits the input file(s) via bus 240 and/or commu-
nication interface 4.0 to an application unit 270. an output
device such as printer 260, or to main memory 250.
The process document module 2.3 includes an extract
data module 2.3.1. This module extracts data off of the 45
document in accordance with the user's instructions, for
example, the user-defined template. or through the interac-
tive mode.
The process document module 2.3 also includes a preap-
plication process module 2.3.2 which gathers and associates 50
information extracted from the document in accordance with
content instructions. This module prompts the user for any
additional information required to satisfy the relationships
defined by the content instructions. The preapplication pro-
cess module 2.3.2 also places the selected information into 55
the transmission format defined by the transmission format
instructions.
The preapplication process module 2.3.2 also generates
the input file 2.3.3 for the selected application in accordance
with the appropriate instructions. The input file 2.3.3 is then 60
transmitted to bus 240 and/or communication interface 4.0
for transmission to a particular application unit 270.
FIG. 10 illustrates a detailed output data flow diagram for
output module 3.0. Output module 3.0 outputs a textual
and/or image copy of the document. In the FIG. 1 preferred 65
embodiment. output module 3.0 is implemented by printer
260. associated software. and associated interface circuitry.
by the correct errors module 2.1.2 and the user. The correc-
tions are reflected in the document information stored in
document file 2.1.4.
The retrieve document module 2.1.3 is used to retrieve
and output a document. The system prompts the user to
specify the storage location of a document and the type of
document copy. for example, a textual or an image copy, to
be outputted. The document is then outputted by the output
process module 3.0.
If the user initially selected maintain definitions
processing. the user would instruct the system to maintain
and/or modify parameter definitions through the maintain
definitions module 2.2. For example. the user can define and
maintain a document template for extracting selected por-
tions of information off of the hard copy document. The user
can use the template to extract selected portions of infor-
mation off of the hard copy document when the document is
originally inputted, or alternatively, the user can use the
template to identify selected portions of information for
extraction off of an image copy of the document. In creating
the template, the user identifies pieces of information on the
document to be extracted and assigns a variable name, or
subfile. to each piece of data.
The location of data to be extracted can be defined in a
number of ways other than by use of a template. For
example, the user can designate the absolute location of
information on the document with respect to a grid overlaid
on the document. e.g., always on line 3, starting in column
1. The user can also identify information by specifying the
relative location of information to be extracted, e.g . always
two lines below the piece of data named "salutation" starting
in column 3. The user can also specify the location of
information to be extracted by variable location
specification- For example. if the hard copy document is a
letter. the module would conduct a key word search for the
term "Dear Sir:". Wherever this term "Dear Sir:" is located,
this piece of data would be associated with the variable
specified by the user, for example. the variable "salutation."
In addition, a defined set of conventional symbols can be
used to signify certain recurring data items for the conve-
nience of users of the instant invention. For example, a "@"
symbol can be used to delineate the vendor name as follows:
"@XYZ Corporation@". Other examples of the use of
symbols to delineate information will be described with
reference to Appendix 4.
The maintain definition module 2.2 is also used to main-
tain data relationships in accordance with content instruc-
Exhibit 1
23
5,625,465
11
tions and to maintain input file formats in accordance with
transmission format instructions. Relationships are defined
and maintained between pieces of data. specified by. for
example. the names of variables. through the define rela-
tionships module 2.2.2. The names of pieces of data on the 5
document are retrieved by. for example. the define template
module 2.2.1. and are passed to the define relationships
module 2.2.2. The user may then provide any additional
pieces of data needed to generate an input file for a particular
application program or unit. such as an input file line 10
number. The user. the applications software. andlor instruc-
tions previously stored in memory then establishes the
contents of the input file by defining relationships between
pieces of data using content instructions. Specific examples
of content instructions will be discussed below in conjunc- 15
tion with Appendices 11. 12A, 12B. 12C 13A. 13E. and
13C
12
The user can also use an application program to process
information by loading the particular application program
into the computer 230 rather than by sending the input file
to a remote application unit 270.
An illustrative example of the processing described above
will now be described.
The user inputs instructions via keyboard 234 or another
input device which indicate that the user desires to input and
store a document. The computer 230 then prompts the user
for the name of the document. In this example. the user
desires to input the document of FIG. 2 and therefore names
the document "XYZ Corp. Bill Dec. 1. 1986." The computer
then prompts the user to feed the hard copy document 100
into the scanner 210. The image of the hard copy document
is displayed on display 232. The computer then prompts the
user to identify the account number on the document. By use
of the mouse 236 or other input device to position a cursor
on the display. the user indicates the location of the account
number. The account number is then read-in to a subfile
The user andlor the applications software defines and
maintains the transmission format of the input file to be used
by a particular application program or unit through the
define format module 2.2.3 in accordance with transmission
format instructions. This is accomplished by defining the
parameters to be used by the preapplication process module
2.3.2 in generating an input file. Parameters which would
typically be required to generate an input file would include
the character type. e.g.. text or pixel; delimiters used
between pieces of data. e.g . a slash or a semicolon; end of
line characters. e.g .. a carriage return or a line feed; and end
20 named "Account Number." This process proceeds until all of
the desired information has been read-in and stored.
In this particular example. no errors were encountered
while inputting the document. The user then directs that the
25 document be stored for future reference in a document file.
of file characters. Examples of transmission formats will be
described in further detail below in conjunction with Appen- 30
dices 11. 12A, 12B. 12C 13A, 13B. and 13C.
Some time later. the user desires to retrieve and output the
document and to generate input files based on information
from the document. The computer 230 prompts the user for
the name of the document and the type of output. The user
responds with "XYZ Corp. Bill Dec. 1. 1986" for a printed
textual copy. The document is then retrieved from the
document file and passed to the printer 260 for printing.
In order to generate an input file for a specific application
program. the user selects the option to define a document
template for use when each month's XYZ Corporation bill
arrives. Accordingly. the user instructs the system to display
a copy of an XYZ Corporation bill on the display 232. The
user then identifies pieces of data by absolute locations. That
is. the user assigns specific names to information located at
If the user initially selected process document processing.
the interface will then proceed to process the document
through use of the process document module 2.3. For
example. the user can extract specific portions of data from 35
an image copy of a document. can generate an input file for
transmission to an application program. or can directly
process information interactively with an application pro-
gram.
40 specific portions of the document. In this example. the user
would input the following information:
If the user desires to extract specific portions of data from
an image copy of a hard copy document which has already
been stored in memory. the user uses the extract data module
2.3.1 to identify a document to be processed. The document
is then retrieved by the retrieve document module 2.1.3 and 45
passed to the extract data module 2.3.1. The user can also
select parameter definitions through the select definitions
module 2.2.4.
The selected document template or parameter definition is
passed to the extract data module 2.3.1. The extract data 50
module 2.3.1 extracts pieces of data from the image copy of
the document. as defined by the document template defini-
tion or the parameter definitions or both. This document data
is then passed to preapplication process module 2.3.2.
The interface generates input file(s) 2.3.3 by use of the 55
preapplication process module 2.3.2. The selected data rela-
tionship definition. as defined by the content instructions.
and the selected record format definitions. as defined by the
transmission format instructions. are passed to the preappli-
cation process module 2.3.2. The preapplication process 60
module 2.3.2 assembles the input file in accordance with the
content instructions. The preapplication process module
2.3.2 also prompts the user for any additional pieces of data
which need to be provided by the user. The input file is
converted to the desired transmission format in accordance 65
with the transmission format instructions. This physically
formatted data is then stored in the input file 2.3.3.
Vendor-text. line 1. one line. column L 80 characters;
Account number-numeric. line 6. one line. column 25.9
characters;
Statement date-date. line 9, one line. column 25. 8 char-
acters;
Payment date-date. line 11. one line. column 25. 8 char-
acters;
Previous balance-currency. line 7. one line. column 75.9
characters;
New charges-currency. line 8. one line. column 75. 9
characters;
Other debits-currency. line 10. one line. column 75. 9
characters;
Finance charges-currency. line 12. one line. column 75. 9
characters;
Payments-currency. line 13. one line. column 75. 9 char-
acters;
Other credits-currency. line 14. one line. column 75. 9
characters;
New balance-currency. line 15. one line. column 75. 9
characters.
The user also identifies data with variable locations. In
this particular example. a variable location is specified as
follows:
Exhibit 1
24
5,625,465
13
Heading 2-line, value="Mail To:"
The identification of Heading 2 as line information means
that the system will search for occurrences of the character
string "Mail To:" and assign the line number which contains
this character string to Heading 2.
The user also identifies data by relative locations. In this
example, the user identifies the following relative location:
Mail To-text. Heading 2+1. 3 lines. column 60. 25 char-
acters per line.
5
The instructions above instruct the system to assign the 10
textual information beginning on one line after Heading 2
and continuing for 3 lines. in column 60, to the Mail To
subfile.
As an alternative to inputting the actual line, column. and
character numbers, the user can identify desired portions of
the document by blocking. or highlighting. the desired 15
portions using the mouse or other input device. In this case,
the computer converts the highlighted portions into corre-
sponding line. column. and character numbers.
Appendix 11 lists data corresponding to the hard copy
document of FIG. 2 and the associated variable or subfile 20
names.
Next, the user desires to define data relationships in
accordance with content instructions. Examples of the type
14
Record number, horizontal position, vertical position,
statement date;
Record number, horizontal position. vertical position.
date received;
payment date;
previous balance;
Record number. horizontal position, vertical position.
new charges;
finance charges;
payments;
new balance.
Next, transmission format instructions are employed to
define the transmission format of the input file for a specific
application program or unit. Appendix illustrates the trans-
mission input file corresponding to Appendix 12A. Appen-
dix FIG. 13B illustrates the transmission input file corre-
sponding to Appendix 12B. Appendix 13C illustrates the
transmission input file corresponding to Appendix 12C. A
comparison of Appendices 12B and 12C reveals that Appen-
of contents which can be specified by a user are illustrated
in Appendices 12A, 12B, and 12C.
In this particular example, three separate departments of
ABC Corporation require information from the XYZ Cor-
poration bill. The first department requires vendor. account
number. statement date, payment date. previous balance,
new charges, debits. finance charges, payments, and new
balance information. The second and third departments
require mail to information and previous balance informa-
tion. Each of these departments have their own application
program which utilizes this information.
25 dices 12B and 12C have the same contents. However, the
information illustrated in FIG. 12B is being sent to a
different application program than the information in Appen-
dix 12C. These application programs require different trans-
mission input formats, as illustrated in Appendices 13B and
30 13C. More specifically, the application program that
receives the input file illustrated in Appendix 13B uses the
greater than sign as a delimiter whereas the application
program which receives the transmission input file shown in
Appendix 13C uses a back-slash as the delimiter.
After the contents and the transmission format for the
input file have been defined, and any additional information
has been inputted, the input file is assembled and transmitted
to the particular application program.
Appendix 14 illustrates another example of a hard copy
40 document containing information to be processed by the
instant invention. The hard copy document illustrated in
Appendix 14 is first scanned and information from the hard
copy document is stored into a memory. The interface 200
then identifies portions of the hard copy document corre-
The user employs content instructions to designate how 35
pieces of information, which have been extracted off of hard
copy document 100, are directed to particular departments,
that is, particular application programs. Appendix 12A illus-
trates the contents of the information to be transmitted to the
first department. FIG. 12B illustrates the information to be
transmitted to the second department. Appendix 12C illus-
trates the information to be transmitted to the third depart-
ment. The content instructions, therefore, parse the infor-
mation shown in FIG. 11 to various application programs, as
shown by Appendices 12A, 12B, and 12C. Content instruc-
tions can also be used to identify additional pieces of data
which are required for the input files of the particular
application programs. In this partiCUlar example, the specific
application programs from the three departments all require
numeric record number information. numeric horizontal
position information, numeric vertical position information,
and date received information. The horizontal and vertical
position information is used by the application program to
specify the location of the received information on a spread-
sheet application program, in this example. The user may
know in advance the content format required by each
application program, that is. in this example. the location
and type of information specified on the spreadsheet. The
user may also employ the split display mode described with
reference to FIG. 3B to generate content format instructions.
Using the content instructions, the user establishes the
following contents for the input file corresponding to Appen-
dices 12A:
vendor;
Record number. horizontal position. vertical position,
account number;
45 sponding to various variables by recognizing a defined set of
symbols. In the Appendix 14 example, triangles delineate
the mailing address, circles delineate the statement date, and
squares delineate the new charges. Information from these
portions of the hard copy document is stored in the corre-
50 sponding memory locations or subfiles for each variable.
The same set of symbols can be used to identify the same
information from one document to the next. Thus, even if the
physical formats of documents are not fixed from one
document to the next, a diversity of hard copy documents
55 can be processed without manually inputting data by rec-
ognition of the defined symbols.
Examples of readily available application programs are
Quicken and Lotus 1,2,3 both of which are widely utilized
in the business community. Quicken. for example is an
60 easy-to-utilize program for writing checks and preparing
business records. Payee. amount and address information
may readily be transmitted from scanner memory 220 and/or
main memory 250 to the Quicken application program for
check writing functions and ledger keeping purposes. Lotus
65 is a well known spreadsheet program which may process
data input into specified cells once this data is placed in
conventional Lotus format.
Exhibit 1
25
15 16
extracts information off of a hard copy document. thereby
eliminating the need to manually input significant amounts
of information from the hard copy document is suitable for
use as an automated digitizing unit. In addition. information
Thus. the instant invention provides an integrated and
comprehensive system for handling information from a hard
copy document. thus permitting a paperless office. In
addition. the invention permits data. extracted off of a hard
copy document. to be easily manipulated into various logical
and transmission formats required by a particular application
unit. The invention also provides a low cost system for
inputting information from a wide variety of hard copy
documents into a memory.
5 can be input by user responses and digital and analog signals
generated from various devices. and from computer files
from other computer systems. Suitable hardware for input-
ting data includes a keyboard. a light pen. a mouse. a touch
screen. a laser scanner. a microphone. a tablet. a disk drive.
The foregoing description has been set forth merely to
illustrate preferred embodiments of the invention and is not
intended to be limiting. Modifications are possible without
departing from the scope of the invention.
10 a magnetic tape drive. and a modem.
The interface 200 can also output information in forms
other than a hard copy of textual or image information. For
example. the interface 200 can output system responses.
computer files. and digital and analog signals for transmis-
15 sion to other computer systems or to control systems.
For example. letters. checks. forms. pictures. reports.
music scores. film. and other types of hard copy documents
can be processed by the invention for accounts payable/
receivable accounting. inventory control. record keeping.
budgeting. data base management. music transcription.
forms processing. computerized art. survey and question-
naire processing. statistical data analysis. correspondence 20
processing and other applications.
Other automated digitizing units can be used in addition
to or as an alternative to use of the scanner 210 as an input
unit. Any electrical. magnetic. or optical device which
Suitable hardware for outputting information includes a disk
drive. a magnetic tape drive. a cathode ray tube. a plasma
screen. a printer. a plotter. a film developer. an amplifier. and
a modem.
Since modifications of the described embodiments incor-
porating the spirit and substance of the invention may occur
to persons skilled in the art. the scope of the invention should
be limited solely with respect to the appended claims and
equivalents.
No.12A
Appendix
(No. 11)
Variable Name
Vendor
Heading 2
Mail To
Acconnt Number
Statement Date
Payment Date
Previous Balance
New Charges
Debits
Finance Charges
Payments
Other Credits
New Balance
Vendor
Acconnt Number
Statement Date
Payment Date
Previous Balance
New Charges
Debits
Value
XYZ Corporation
2
XYZ Corporation
PO Box 567
Anywhere, NY 63130
123456789
12101186
1101187
$1234.56
$789.D1
$2.34
$1000.00
$1025.91
XYZ-Corporation
123456789
12101186
1101187
$1234.56
$789.D1
Finance Charges $2.34
Payments $1000.00
New Balance $1025.91
No. 12B Mail To XYZ Corporation
PO Box 567
Anywhere, NY 63130
Previous Balance $1234.56
No. 12C Mail To XYZ corporation
PO Box 567
Anywhere, NY 63130
No. 13A >1>1>1>"XYZ Corporation"
>2>2>25>+123456789>
>3>2>I>DI2101186>
>4>2> 11>DI21 15/86>
>5>2>21>D01l01l87>
>6> 10>25>$1234.56>
>7>11>25>$789.01>
>8>13>25>$2.34>
>9>14>25>$1000.00>
>10>16>25>$1025.91>
No. 13B >1>1>1>$1234.56>
Exhibit 1
26
5,625,465
No.13C
Q
\.20
Customer
ABC Corporation
Hometown, NJ 88981
Account Number 123456789
17
-continued
>2>2>I>"XYZ Corporation"
>3>3>1>"PO Box 567"
>4>4>I>"Anywhere, NY 63130"
11/111/$1234.5611
12I2II/*XYZ Corporation*
13/3/1/*PO Box 567*
14/4/11* Anywhere, NY 63130*
Appendix
No. 14
XYZ Corporation
Mail To:
XYZ Corporation
PO Box 567
Anywhere, NY 63130
$1234.56
18
Statement Date: 12181186
1181187
Previous Balance
New Charges
Other Debits
$789.01
Payment Date:
Finance Charges (10%)
Payments
$2.34
$1888.88
Other Credits WINIER SALE IN EFFECT 30
THROUGHOUT JANUARY! New Balance $1825.91
What is claimed is:
1. A multimode information processing system for input-
ting infonnation from a hard copy document into at least one
application program. comprising:
30
(a) an automated digitizing unit providing digitized image
information representative of an image of at least a 35
portion of said document;
(b) a display to display information;
(c) an input device to receive user instructions from a
user; and
(d) a computer. operable with at least one of the auto- 40
mated digitizing unit. the display, and the input device;
said system having character recognition capabilities for
recognizing at least a portion of alpha/numeric charac-
ters appearing on said document for providing textual
data. said digitized image information and said textual 45
data collectively comprising document infonnation.
and said system configured to route at least a portion of
said document infonnation as input data to be pro-
cessed by at least one application program to said at
least one application program. and to operate in at least 50
one of:
(1) an interactive mode wherein at least a portion of
said document information is displayed on the dis-
play and the input device conveys instructions from
the user for locating information required by the 55
application program within said document informa-
tion and for transferring said located information to
at least one field of infonnation required by said at
least one application program;
(2) an interactive template creation mode wherein an 60
image representative of at least a portion of said
document is displayed on the display and the input
device conveys instructions from the user to create
and store in said system a template which identifies
a location within said document infonnation of at 65
least one field of infonnation required by said at least
one application program; and
(3) an automatic mode to match at least a portion of
said document infonnation with a template created
or compatible with said interactive template creation
mode stored in said system to extract fields of
infonnation required by the application program and
to transfer said fields of information to said at least
one application program;
wherein the display projects an image showing at least a
portion of said document information and at least one field
of information required by said at least one application
program during the interactive mode.
2. A multimode information processing system as set forth
in claim 1. wherein said system further comprises:
a memory for storing a defined set of symbols which
designate fields of information required by said at least
a symbol detection means for detecting the presence of a
particular one of said defined set of symbols within said
document information and for extracting a field of
information required by said at least one application
program based on said detecting.
3. A multimode information processing system as set forth
in claim 1 wherein said system is operative in said interac-
tive mode for locating said at least one field of information
required by said at least one application program by block-
ing portions of said at least a portion of said document
information displayed on the display using the input device.
4. Amultimode information processing system as set forth
in claim 1 wherein said system is operative in said interac-
tive template creation mode for creating a template by
blocking portions of said at least a portion of said document
in claim 1. further comprising:
a memory for storing digitized image infonnation repre-
sentative of the actual appearance of at least a portion
of said hard copy document.
Exhibit 1
27
5,625,465
19
20
information required by the application program and
a memory for storing at least a portion of said textual data.
a memory for storing digitized image information repre-
of said hard copy document; and
5 portion of said document information and at least one field
program during said interactive template creation mode.
8. Amultimode information processing system as set forth 10
in claim 1. wherein said character recognition capabilities
are included in the automated digitizing unit.
in claim 1 wherein in said interactive template creation
mode. said template identifies a plurality of locations within 15
said document information of a plurality of fields of infor-
mation required by an application program.
10. A multimode information processing system as set
forth in claim 1 wherein said at least one application
program is a plurality of application programs. 20
forth in claim 1 wherein in said interactive template creation
mode. said template identifies a plurality of locations within
said document information of a plurality of fields of infor-
mation required by a plurality of application programs. 25
12. Amultimode information processing system for input-
ting information from a hard copy document into at least one
(a) an automated digitizing unit providing digitized image 30
information representative of an image of at least a
user; and
(d) a computer. operable with at least one of the auto-
mated digitizing unit. the display. and the input device;
35
ters appearing on said document for providing textual 40
data. said digitized image information and said textual
data collectively comprising document information.
said document information as input data to be pro-
cessed by at least one application program to said at 45
least one application program. and to operate in at least
one of:
play and the input device conveys instructions from 50
the user for locating information required by the
at least one field of information required by said at
least one application program; 55
(2) an interactive template creation mode wherein an
and store in said system a template which identifies 60
a location within said document information of at
least one field of information required by said at least
said document information with a template created 65
forth in claim 12 wherein said system is operative in said
interactive template creation mode for creating a template by
user; and
. (d) a computer. operable with at least one of the auto-
one of:
said document information with a template created
wherein. in said interactive mode. said located information
is located for and transferred to at least two fields.
user; and
Exhibit 1
28
5,625,465
21
(d) a computer, operable with at least one of the auto-
mated digitizing unit, the display, and the input device;
data, said digitized image information and said textual
data collectively comprising document information,
cessed by said at least one application program to said 10
at least one application program, and to operate in at
least two of:
(2) an interactive template creation mode wherein an 20
a location within said document information of at 25
30
22
cessed by said at least one application program to said
least two of:
and store in said system a template which idelltifies
wherein said computer is operative in said interactive mode to transfer said fields of information to said at least
35 for locating said at least one field of information required by
said at least one application program by blocking portions of
said at least a portion of said document information dis-
played on the display using the input device.
forth in claim 15 wherein said system further comprises:
40
application program, comprising:
a symbol detection means for detecting the presence of a 45
17. A multimode information processing system as set 50
forth in claim 15 wherein said computer is operative in said
interactive mode for locating said at least one field of
information required by said at least one application pro-
gram by blocking portions of said at least a portion of said
document information displayed on the display using the 55
input device.
user; and
65
user; and
least two of:
Exhibit 1
29
5,625,465
23
15
20. A multimode information processing system as set 20
forth in claim 19 wherein said computer is operative in said
interactive template creation mode for creating a template by
21. Amultimode information processing system for input- 25
user; and
30
at least one application program. and to operate in at 45
least two of:
24
wherein said computer is operative in said interactive tem-
plate creation mode for creating a template by blocking
portions of said at least a portion of said document infor-
mation displayed on the display using the input device.
user; and
at least one application program. and to operate in at
least two of:
further comprising a memory for storing digitized image
information representative of the actual appearance of at
least a portion of said hard copy document.
55 23. Amultimode information processing system for input-
a location within said document information of at 60
or compatible with said interactive template creation 65
user; and
Exhibit 1
30
5.625,465
25
said document information as input data to be pro- 5
least two of:
said document information is displayed on the dis- 10
at least one field of information required by said at 15
device conveys instructions from the user to create 20
(3) an automatic mode to match at least a portion of 25
to transfer said fields of information to said at least 30
further comprising a memory for storing at least a portion of
said textual data.
ting information from a hard copy document into at least one 35
user; and
40
mated digitizing unit. the display. and the input device; 45
data collectively comprising document information. 50
least two of: 55
application program within said document informa- 60
26
further comprising:
forth in claim 24 wherein said character recognition capa-
bilities are included in the automated digitizing unit.
user; and
least two of:
wherein, in said interactive mode. said located information
65 is located for and transferred to at least two fields.
forth in claim 26 wherein in said interactive template
Exhibit 1
31
5,625,465
27
creation mode, said template identifies a plurality of loca-
tions within said document information of a plurality of
fields of information required by an application program.
forth in claim 26 wherein said at least one application 5
program is a plurality of application programs.
forth in claim 26 wherein in said interactive template
creation mode, said template identifies a plurality of loca-
tions within said document information of a plurality of 10
fields of information required by a plurality of application
programs.
15
(c) an input device to receive user instructions from a 20
user; and
said system having character recognition capabilities for 25
and said system configured to route at least a portion of 30
least one application program, and to operate in each
of:
28
user; and
of:
application program within sind document informa-
(1) an interactive mode wherein at least a portion of 35
40 of information required by said at least one application
document is displayed on the display and the input 45
said document information with a template stored in
said system to extract fields of information required
by the application program and to transfer said fields
50
of information to said at least one application pro- 55
gram; and
forth in claim 30 or claim 31, wherein said system further
comprises:
a symbol detection means for detecting the presence of a
forth in claim 30 or claim 31, further comprising:
of said hard copy document.
60 forth in claim 30 or claim 31, further comprising:
forth in claim 30 or claim 31, wherein said character
recognition capabilities are included in the automated digi-
tizing unit.
forth in claim 30 or claim 31. wherein in said interactive
Exhibit 1
32
5,625,465
29
template creation mode. said template identifies a plurality
of locations within said document infonnation of a plurality
of fields of infonnation required by an application program.
forth in claim 30 or claim 31. wherein said at least one 5
application program is a plurality of application programs.
forth in claim 30 or claim 31. wherein in said interactive
template creation mode. said template identifies a plurality
of locations within said document infonnation of a plurality 10
of fields of information required by a plurality of application
programs.
39. Amultimode infonnation processing system for input-
infonnation representative of an image of at least a
15
(b) a display to display infonnation;
user; and
data. said digitized image infonnation and said textual
least one application program. and to operate in each
of:
30
user; and
of:
said document infonnation is displayed on the dis-
the user for locating infonnation required by the
application program within said document infonna-
tion and for transferring said located infonnation to
a location within said document infonnation of at
said document infonnation with a template created
infonnation required by the application program and
to transfer said fields of infonnation to said at least
said document infonnation is displayed on the dis-
the user for locating infonnation required by the
application program within said document infonna-
tion and for transferring said located infonnation to
40 wherein said computer is operative in said interactive mode
for locating said at least one field of infonnation required by
said at least a portion of said document infonnation dis-
document is displayed on. the display and the input
.device conveys instructions from the user to create
45 41. A multimode infonnation processing system as set
forth in claim 30 or claim 31, further comprising:
a location within said document infonnation of at
said document infonnation with a template stored in
said system to extract fields of infonnation required
50
of infonnation to said at least one application pro- 55
gram; and
wherein said computer is operative in said interactive mode
for locating said at least one field of infonnation required by
said at least a portion of said document infonnation dis- 60
40. Amultimode infonnation processing system for input-
a memory for storing digitized image infonnation repre-
42. A multimode infonnation processing system for input-
user; and
Exhibit 1
33
5,625,465
31
of: 5
32
10 wherein the display projects an image showing at least a
15 ting information from a hard copy document into at least one
least one field of information required by said at least 20
by the application program and to transfer said fields 25
of information to said at least one application pro-
gram; and
of information required by said at least one application 30
user; and
recognizing at least a portion of alpha/numeric charac- 45
said document information as input data to be pro- 50
of:
user; and
of:
gram; and
said document information is displayed on the dis- 55
wherein said system is operative in said interactive template
creation mode for creating a template by blocking portions
60 of said at least a portion of said document information
displayed on the display using the input device.
device conveys instructions from the user to create 65
Exhibit 1
34
5,625,465
33
user; and
mated digitizing unit, the display, and the input device; 5
10
~ 0
34
gram; and
wherein, in said interactive mode, said located information
20 is located for and transferred to at least two fields.
25
least one field of information required by said at least 30
35
wherein said system is operative in said interactive template
40
creation mode for creating a template by blocking portions
of said at least a portion of said document information
displayed on the display using the input device.
45
user; and
50
of:
user; and
of:
wherein, in said interactive mode, said located information
is located for and transferred to at least two fields.
Exhibit 1
35
5,625,465
35
user; and
mated digitizing unit. the display, and the input device;
10
cessed by at least one application program to said at 20
one of:
36
least two of:
play and the input device conveys instructions from 25
tion and for transferring said located information to wherein the display projects an image showing at least a
30 of information required by said at least one application
program during at least said automatic mode.
program during said automatic mode.
45
49. Amultirnode information processing system for input- 50
user; and
55
50. A multirnode information processing system for input-
user; and
of:
Exhibit 1
36
5,625,465
37
said system to extract fields of information required 5
gram; and
portion of said document information and at least one field 10
program during at least said automatic mode.
. information processing system for input-
ting mformatIOn from a hard copy document into at least one
15
user; and
of:
tion and for transferring said located information to 40
38
user; and
one of:
document is displayed on the display and the input 45
wherein said system is operative in said interactive mode for
said at field of information required by
50 sllld at least one application program by blocking portions of
said at least a portion of said document information dis-
information required by the application program and 55
of information required by said at least one application 60
program during said automatic mode.
forth in claim 30 or claim 31. wherein said computer is
operative in said interactive mode for locating said at least
one field of information required by said at least one
application program by blocking portions of said at least a
portion of said document information displayed on the
display using the input device.
* * * * *
Exhibit 1
37
United States Patent [19]
Lech et al.
[54] INFORMATION PROCESSING
METHODOLOGY
[75] Inventors: Robert Lech. Jackson; Mitchell A.
Medina. Essex Fells; Catherine B.
E6as. Plainsboro. all of N.J.
[73] Assignee: Millennium L.P .. Grand Cayman,
Cayman Islands
[21] Appl. No.: 487,150
[22] Filed: Jon. 7,1995
[62]
[51]
[52]
[58]
[56]
Related U.S. App6cation Data
Division of Ser. No. 348,224, Nov. 28, 1994, Pat No.
5,625,465, which is a continuation of Sec. No. 143,135, Oct
29, 1993, Pat. No. 5,369,508, which is a continuation of Sec.
No. 672,865, Mar. 20, 1991, Pat. No. 5,258,855.
Int. Cl.
6
...................................................... G06F 15/40
U.S. CI .............................................. 3821180; 382/175
Field of Search ..................................... 3951148. 149;
4,760,606
4,802,104
4,931,957
5,031,121
5,228,100
5,367,619
5,404,294
382/287,175,306.177.180.282.317
References Cited
U.S. PATENT DOCUMENTS
7/1988 Lesnick et aI. ........................... 382/48
111989 Ogiso ...................................... 3641518
6/1990 Takagi et aI ............................ 3641521
711991 Iwai et al ................................ 3641523
7/1993 Takeda et aI ............................. 382161
1111994 Dipaolo ................................... 395/149
4/1995 Kamik ................................. 3641419.1
I11111 1111111111111111111111111111111111111111111111111111111111111
US005768416A
[11] Patent Number: 5,768,416
Jun. 16, 1998 [45] Date of Patent:
5,416,849
5,448,738
5,452,379
5,506,697
5/1995 Huang ..................................... 3821173
9/1995 Good et aI .............................. 3951700
9/1995 Poor ........................................ 3821317
4/1996 Ii et aI. .. ................................ 358/448
OTHER PUBUCATIONS
Que's Computer User's Dictionary. 2nd Ed. Bryan Pfaffen-
berger (author); 1991; p. 144.
Primary Examiner-Edward L. Coles. Sr.
Assistant Examiner-Stephen Brinich
Attome}\ Agent, or & Lardner
[57] ABSTRACT
An infonnation processing methodology gives rise to an
application program interface which includes an automated
digitizing unit, such as a scanner, which inputs information
from a diversity of hard copy documents and stores infor-
mation from the hard copy documents into a memory as
stored document infonnation. Portions of the stored docu-
ment information are selected in accordance with content
instructions which designate portions of the stored document
infonnation required by a particular application program.
The selected stored document information is then placed
into the transmission format required by a particular appli-
cation program in accordance with transmission format
instructions. After the information has been transmission
fonnatted. the information is transmitted to the application
program. In one operational mode. the interface interactively
prompts the user to identify, on a display. portions of the
hard copy documents containing information used in appli-
cation programs or for storage.
66 Claims, 15 Drawing Sheets

10
I x:fZ Corporation fJ
10
r'
Customer Moil To'
ABC Corporation 10 XYZ Corporation
i-' PO Box 567
Hometown.NJ 08901
Anywhere,NY 63130
-,
.-____ __
I Account Number 1123456789
10
10
r Previous Balance
r
Statement Dote:
Payment Dote:
10./
r"
12/01/86
1101187
10
10
,r New Charges
, ... Other Debits
10
I ..... Finance Chorges(lO"lo)
10 10
j
1- Payments
Other Credits
",
New Balance
10
'I
10
Winter Sole In Effect Throughout January!
30)
10
r'
S1234.56
S789.01

$2.34
5'
SIOOO.OO

S1025.91
o
P
Exhibit 2
38
u.s. Patent
Jun. 16, 1998 Sheet 1 of 15 5,768,416
z
6"
b
-!"'
c:
" ,... c:
~
,
0 0 0
&()
-
to
-
0
:;::
I"-
:;::
cO
CD
O- N e-
N
C\J
-
I"-
u- u-
-e
c:
(\I
._ c

._ c
CD ...
a . ~ Q . ~
Ee
a..
a. 0.
<{ <{
0
~
,
NI
,
N
0
r<')
v
~ 1<)1""'"
'}'
r-rt")
_(\I
/
N
::l
a.
E
8---
\ O ~
-
-
~
...
0
0
E
N
~
CD
~
E
0
0
N
...
Q
CD
N
c:
l-.J
c
0
u
U)
a
a
~ -
Exhibit 2
39
u.s. Patent Jun. 16, 1998 Sheet 2 of 15
,...--------, 10
I XYZ Corporation ~
Customer ~ O
~ - - - - - - - - - - - - - - - - ~
ABC Corporation 10
123 Sixteenth Street ~
Hometown, NJ 08901
c
lO
10
I Account Number 1123456789 riO
10 10 10
r
F
r
./'
Statement Date: 12/01/86 \..-
10
10
Payment Date: 1101187
10../ j
10 10
\ ......
, .....
./
10
Mail To:
XYZ Corporation
PO Box 567
Anywhere, NY 63130
"-10
Previous Balance
New Charges
Other Debits
Finance Charges {lO%)
Payments
Other Credits
New Balance
\...
10
Winter Sale In Effect Throughout January!
30)
FIG. 2
5,768,416
10
'
$1234.56
$789.01
$2.34
SIOOO.OO
$1025.91
~ O
10
~
10
...)
~
o
o
~ -
I o
~
Exhibit 2
40
u.s. Patent
Jun. 16, 1998
Sheet 3 of 15
5,768,416
f 1
.,
o i
-
Exhibit 2
41

6)
I XVZ CORPORATION I
CUSTOMER MAIL TO:
ABC CORPORATION XYZ CORPORATION
123 SIXTEENTH STREET POBOX 567
HOMETOWN, NJ 08901 ANYWHERE, NY 63130
IACCOUNT NO. \123456789 I

$1234.56
ISTATEMENT DATE 112101/87 1

$789.01
fPAYMENT DATE I 1/01/87 J
OTHER
DEBITS

$2.34
PAYMENTS $1000.00
WINTER SALE IN EFFECT

DTS
THROUGHOUT JANUARY I
NEW
$1825.91
BALANCE
l
232L
FIG. 38
ACCOUNTS PAYABLE
r
232C
VENDOR r:::J
INVOICE II
INVOICE DATE
DATE DUE
AMOUNT DUE
I

230

00

.....

a

I-'
,,1;1\
I-'

g:
a

I-'
til
til
"..
-....)
0'\
QO
"..

0'\
Exhibit 2
42
u.s. Patent Jun. 16, 1998
Sheet 5 of 15
5,768,416
User
Dialog
(1.0
2.0
3.0)
Input Output
Input
Document
Info
Document
Output
Process
Process Process
Transmission
4.0
~
Comm. -
External
Device
FIG. 4
Exhibit 2
43
u.s. Patent Jun. 16, 1998
2\
en
en
Oal
-u
Co
-\0.,
a..
c
-Q,)
~ E
C>.::J
c:u
-0
a
->--
)(0.
~ 8
-
-
rt)
.
-
..,;-
a:
c::l
.&:.Q.
U
u.s
Sheet 6 of 15
0
(7
Q)
0" >-
00.
EO
-u
C\J
--:"
<V_
O I ~
Co.
E.E
o
-
\0., C\J
~ V
u
(/)
5,768,416
0
~
E
E
0
u
E
e
LL
10

(.9
-
lJ..
Exhibit 2
44
Input
Process
1.0
Input
Document
User
Dialog
2.1
Output
Mntn Document
Library
Document Document
2.2
---r Definitions
Process
r-i Doc
Mntn
oef's
Dialog Dialog
Transmission
3.0
Comm. 1
Dialog I I Dialog 4.0
FIG. 6
External
Device
Output
Process
3.0
,
e

r.Il

=
....

a

....

....

00
g

'"-I

....
Ut
VI
".
....J
0\

".

0\
Exhibit 2
45
1.0
Input
Process
Document
File
3.0
J
Input
Document
2.1.1 I 1
r Document Document
f
Output
Retr Document
Store h (\ ,
Output
Process
Errors
2.1.2
'--
FIG. 7
Doc
Correct
Errors
Dialog Diolog

Dialog
'- User
l-'
Doc
!Document
Document
r 1 Y2
..
Mntn
Oef's
2.1
I--'
2.3
t
Process
Doc
Cj

rJl

;F
""""
a

....

....

co

co
S,
....
Ul
01

QC
.,.
.&;;..

Exhibit 2
46
2.1
2.2.1)
Input
Mntn
Document
Define
Library
Tempi
Document
Selective
Variable
Template
Scan Definition
Instructions
Dialog
)
Content
2
f
2
Data
Format
Relationship
User
Inst[
Define
Definition
Dialog
ReI
Transmission ~ Dialog
Format
Physical Instructions
223)
Format
Define
Definition
Format
FIG. 8
2 . 2 ~ Definitions
\
1
r--
Select
Def's
Dialog
Definitions
21
Definition
File
t-
2
.
2
./
2 . 3 ~
Process
Doc
)
User
~

00

;p
....
g
....
~
~
~ a - . .
~
~
~
~
\0
o
~
~
Ut
tit
---
.....:a
="
Q()
--- ~
~
Q\
Exhibit 2
47
2.1
- ~
2.2
s....
Mntn
Library
Mntn
Oet's
Document
Selective
Scan
Information
~
Content
Format
Jnformation
~
/
Transmission
Format
Information
___ -----ir2.3.l
Extroct
Data
2.3.2
1
Dialog
Document
Data
Dialog
User
...)
To Bus 240
Pre AppJ ] (
Process I )
2.3.3
'--
FIG. 9
Input
File
Comm
./
W,
O
2.3
I---'
~

00

~
=
......
~
a
~
~
9'
~
~
rJ:J
=-
m.
~
Q
~
~
til
OJ
...
.....:J
="
QC
...
...
~
="
Exhibit 2
48
u.s. Patent
>-
- ~
Xo
J!?u
Jun. 16, 1998
ell)
c:u
.... -
CD>
-c
"'0
I.LJ
o __ ""'--_
q:\..
-
_c
::;,CD
~ E
-::;,
::;'u
00
E
E
o
u
'\
en
Sheet 11 of 15
. E g J ~
c: (,) 0
_0
'- .
a. ru
o
!'til
5,768,416
o
-
.
C!)
-
LL.
Exhibit 2
49
u.s. Patent JUD. 16, 1998 Sheet 12 of 15 5,768,416
FIG. 11
r
~ - - - - - - - - - - - - - - ~ A ~ - - - - - - - - - - - - - - ~
\
Variable Name
Vendor
Heading 2
Mail To
Account Number
Statement Date
Payment Date
Previous Balance
New Charges
Debits
Finance Charges
Payments
Other Credits
New Balance
Value
XYZ Corporation
2
XYZ Corporation
PO Box 567
Anywhere, NY 63130
123456789
12101/86
1/01/87
$1234.56
$789.01
$2.34
$1000.00
$1025.91
Exhibit 2
50
u.s. Patent Jun. 16, 1998 Sheet 13 of 15 5,768,416
FIG.12A
(
A
\
Variable Name Value
Vendor XYZ Corporation
Statement Date 12101/86
Payment Date 1/01/87
New Charges $789.01
Debits
Finance Charges $2.34
Payments $1000.00
FIG. 128
(
A.
\
Variable Name Value
Mail To XYZ Corporation
PO Box 567
Anywhere, NY 63130
FIG.12C
__ - - - - - - - - - - - - - - ~ A ~ - - - - - - - - - - - - - - - -
r "\
Variable Name
Mail To
Previous Balance
Value
XYZ Corporation
PO Box 567
Anywhere, NY 63130
$1234.56
Exhibit 2
51
u.s. Patent Jun. 16, 1998 Sheet 14 of 15
FIG.13A
FIG.13B
FIG.13C
> 1 > 1 >1 > "XYZ Corporation"
> 2 > 2 > 25 >+ 123456789 >
> 3 > 2 > 1 > D121 01 186 >
> 4 > 2 > 11 > D12/151 86 >
> 5 > 2 > 21 > D011 01187 >
> 6 > 10 > 25 > $1234.56 >
> 7 > 11 > 25 > $789.01 >
> 8 > 13 > 25 > $2.34 >
> 9 > 14> 25 > $1000.00 >
> 10 >16> 25 > $1025.91 >
> 1 > 1 >1 > $1234.56 >
> 2 > 2 > 1 > IIXYZ Corporation"
> 3 > 3 > 1 > "PO Box 567"
> 4> 4 > 1 > "Anywhere, NY 63130
11
11/1/11 $1234.561/
12/2/11 *XYZ Corporation*
131 311 1 * PO Box 567*
14/4/11 *Anywhere, NY 63130*
5,768,416
Exhibit 2
52
u.s. Patent Jun. 16, 1998 Sheet 15 of 15 5,768,416
FIG. 14
~ ______________________ ~ A ~ ________________________ ~
( \
Customer
ABC Corporation
Hometown I NJ 88981
Statement Date: 0 12101/860
Payment Date: 1/01/87
XYZ Corporation
WINTER SALE IN EFFECT THROUGHOUT JANUARY:
30)
Mail To:
Do XYZ Corporation
PO Box 567
Anywhere, NY 63130 D.
New Charges 0$789.010
Other Debits
Finance Charges (10%} $2.34
Payments $1000.00
Other Credits
Exhibit 2
53
5,768,416
1
INFORMATION PROCESSING
METHODOLOGY
2
can read information off of checks having a rigidly defined.
or fixed, format. However, as the diversity of forms received
by a business increases relative to the number of forms that
must be processed, it becomes less cost-effective to design
This application is a divisional of application Ser. No.
081348,224, filed Nov. 28, 1994, now U.S. Pat. No. 5,625,
465, in turn a continuation application of Ser. No. 081143,
135, filed Oct. 29, 1993. now U.S. Pat. No. 5369,508, and
which is in turn a continuation of application Ser. No.
07/672.865, filed Mar. 20, 1991. now U.S. Pat. No. 5,258,
855.
5 a dedicated machine for processing each type of form
format. This problem is particularly significant in smail
businesses which may, for example, receive fifty invoices a
month, all in different. non-fixed, formats. It is frequently
not cost-effective for a small business to design dedicated
BACKGROUND OF THE INVENTION
10 systems for inputting information in each of these various
formats. This leaves a small business with no other practical
alternative than to manually input the information off of each
invoice each month.
The invention is directed to a system for efficiently
processing information originating from hard copy docu-
ments. More specifically, the invention is directed to a hard 15
copy document to application program interface which
minimizes the need to manually process hard copy docu-
ments.
In the past, information contained on hard copy docu-
ments was manually entered into a computer via the input 20
controller of a particular computer. The original document
was then filed away for future reference. Automatic input of
data was limited to the input of Magnetic Ink Character
Recognition (MICR) data and to Optical Character Recog-
nition (OCR) data. This fixed-position data was forwarded 25
directly to a dedicated computer application specifically
designed to accommodate the input format. In more recent
years, typewritten text has been mechanically inputted into
a computer via a text file. Examples of this latter type of
system are word processors and photo-typesetters. 30
These conventional systems have limitations which
decrease the efficiency of processing information from a
hard copy document. For example, the systems discussed
above are limited in their application to MICR. OCR, or
35
typewritten data. Parsing and processing data is limited to
the particular requirements of the particular computer appli-
cation which requires the input data. In addition, in these
conventional systems, the actual hard copy document must
be retained for future reference at great expense.
In a sophisticated computer network. different users may
40
SUMMARY OF THE INVENTION
It is an object of the invention, therefore, to provide an
application program interface which allows a user to select
specific portions of information extracted from a di versity of
hard copy documents and allows the user to direct portions
of this information to several different users in accordance
with the needs of the particular user.
It is also an object of the invention to provide a cost-
effective system for inputting hard copy documents which
can accommodate hard copy documents in a diversity of
formats.
It is another object of the invention to provide an appli-
cation program interface which allows a user to put
information, which is to be transmitted. into a particular
transmission format. based upon the needs of the receiver of
the information.
It is a further object of the invention to provide an
application program interface which will allow the
extraction, selection, formatting, routing. and storage of
information from a hard copy document in a comprehensive
manner such that the hard copy document itself need not be
retained.
It is another object of the invention to provide a system
which reduces the amount of manual labor required to
process information originating from a hard copy document.
A further object of the invention is to reduce the time
required to process information originating from a hard copy
document so that a higher volume of transactions involving
hard copy documents can be processed.
The invention provides an application program interface
which inputs a diversity of hard copy documents using an
automated digitizing unit and which stores information from
the hard copy documents in a memory as stored document
require different portions of the information contained on a
hard copy document. For example, if the hard copy docu-
ment is an invoice returned with payment of a bill, the
accounting department may need all of the monetary infor- 45
mation contained on the bill while the mailroom may need
only customer address information. to update a customer's
address. Therefore, there is a need for a system in which
specific information from a hard copy document can be
selectively distributed to various users.
50 information. Portions of the stored document information
are selected in accordance with content instructions which
define portions of the stored document information required
by a particular application unit. Selected stored document
information is then formatted into the transmission format
Another problem with conventional systems is that users,
even within the same company, may require that the infor-
mation extracted from a hard copy document be transmitted
to a particular application program in a specific transmission
format. For example, one department in a company may use 55
a particular application program which must receive infor-
mation using a particular character as a delimiter and other
departments may require the information in a different
format using different delimiters.
Another problem, particularly for small businesses, is that 60
current systems can not efficiently accommodate the input-
ting of information from a diversity of hard copy documents.
A large business which receives many forms in the same
format can afford a system which inputs a high volume of
information in that format into memory. For example, it is 65
cost-effective for a bank which processes hundreds of thou-
sands of checks a month to buy a dedicated machine which
used by the particular application program based on trans-
mission format instructions. The transmission formatted
selected stored document information is then transmitted to
the particular application program. The hard copy docu-
ments may contain textual information or image information
or both.
The interface operates in three different modes.
In a first mode, the interface extracts all of the information
from hard copy documents and stores this information in
memory. Parsing of various portions of the extracted infor-
mation is performed in accordance with content instructions.
In a second mode. the user operates interactively with the
interface by use of a display and an input device, such as a
Exhibit 2
54
5,768,416
3
mouse. In this second mode. a hard copy document is
inputted and displayed on the display. The intetface then
prompts the user to identify the location of various infor-
mation. For example, the ntetface can ask the user to identify
the location of address information on the hard copy docu- 5
ment. In response. the user positions the mouse to identify
address information using a cursor. The identified informa-
tion is then stored as address information in memory.
Subsequently. the intetface again prompts the user to iden-
tify other pieces of information. which are then stored in the 10
appropriate locations in memory. This process proceeds until
all of the information which is desired to be extracted off of
the hard copy document is stored in memory.
In a third mode of operation. selected portions of infor-
mation are extracted off of hard copy documents in accor-
dance with predetermined location information which has 15
been specified by the user. For example. the user can define
a template which specifies the location of information on
hard copy documents. Templates can be formed in conjunc-
tion with second mode operation. Alternatively, the user can
instruct the interface to search hard copy documents for a 20
particular character or symbol. located on the hard copy
documents. The information desired to be extracted off of
the hard copy documents is specified relative to the location
of this character or symbol.
The intetface can also prompt or receive from an appli- 25
cations program or another information processing system.
required information. content instructions, and format
instructions.
Other objects, features. and advantages of the invention
will be apparent from the following detailed description of 30
the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
4
FlG. 14 illustrates another example of a hard copy docu-
ment containing information to be processed by the instant
invention.
DErAILED DESCRIPTION OF THE
PREFERRED EM:BODIMENTS
Hardware
The invention provides an intetface between information
originating from a hard copy document and a computer
application unit which uses the information. The computer
application unit can be a particular computer application
program or a device which is controlled in accordance with
instructions or information from the hard copy document.
The invention also allows storing a copy of the hard copy
document in a memory and retrieving the copy of the hard
copy document. By providing a comprehensive and inte-
grated system which can accommodate almost all of the
possible uses of information contained on a hard copy
document, the instant invention allows for a paperless office.
The invention includes hardware and software necessary
to extract, retrieve, and process information from the hard
copy document. A copy of the actual image of the hard copy
document is stored in memory. Textual information
extracted from the hard copy document is also stored in
memory. Textual information is information. such as alpha-
numeric characters. which is recognized on the hard copy
document and which is stored in a form which corresponds
to the particular recognized character. For example. the
extracted characters can be stored in the ASClI format in an
electronic memory.
The user can have all of the information extracted from
The invention will be described in further detail below the hard copy document and stored in memory.
Alternatively. the intetface can interactively prompt the user
with reference to the accompanying drawings, in which: 35
to identify specific pieces of information for storage. The
FlG. 1 illustrates hardware for implementing a preferred intetface can also extract specific pieces of information
embodiment of the instant invention; using a predefined template. The intetface can also prompt
FlG. 2 illustrates an example of a hard copy document or receive from another information processing system or an
containing information to be processed by the instant inven- applications program desired information. content
tion; 40 instructions, and format instructions.
FlGS. 3A and 3B are enlarged views of the computer of
FlG. 1 used to explain how the invention interactively
prompts a user to identify information;
FlG. 4 is an overall data flow diagram for the FlG. 1
preferred embodiment;
FlG. 5 is a detailed input data flow diagram for the FlG.
FlG. 6 is a detailed information processing data flow
diagram for the FlG. 1 preferred embodiment;
45
The instant invention also provides for parsing informa-
tion extracted from the hard copy document and for direct-
ing this parsed information to specific users or application
programs as an input file.
The invention also permits the user to define the trans-
mission format of the input file for a particular computer
application unit.
FlG. 7 is a more detailed information processing data flow 50
FlG. 1 illustrates hardware for implementing a preferred
embodiment of a hard copy document to application pro-
gram intetface according to the instant invention. The inter-
face 200 processes information extracted off of hard copy
document 100 and provides information to application units
270 in a form required by each particular application unit.
diagram for the maintain library module of FlG. 6;
FlG. 8 is a more detailed information processing data flow
diagram for the maintain definitions module of FlG. 6;
FlG. 9 is a more detailed information processing data flow 55
diagram for the process document module of FlG. 6;
The intetface extracts information off of a hard copy docu-
ment 100 utilizing a scanner 210. The scanner 210 can be
FlG. 10 is a detailed output data flow diagramfortheFlG.
FlG. 11 lists data corresponding to the hard copy docu-
ment of FlG. 2;
FlGS. 12A, 12B, and 12C illustrate examples of data
which can be selected from the extracted data of FlG. 11 in
accordance with content instructions;
FlGS. 13A. 13B, and 13C illustrate examples of the data
of FlGS. 12A. 12B. and 12C formatted in accordance with
various transmission format instructions to form input files;
and
any type of scanner which extracts information off of hard
copy documents. for example, an optical Reader.
The scanned information is stored in a scanner memory
60 220 or in main memory 250. as will be described in greater
detail below. If main memory 250 or another memory is
available to store the scanned information, then scanner
memory 220 can be omitted.
The information from scanner memory 220 or main
65 memory 250 is transmitted to computer 230. In the preferred
embodiment. computer 230 includes a display 232. a key-
board 234. and a mouse 236. The display 232 displays an
Exhibit 2
55
5,768,416
5
image of the hard copy document itself and/or information
necessary to process the information extracted off of the hard
copy document.
The computer 230 is used to select portions of the stored
document information contained in memory in accordance 5
with content instructions which define portions of the stored
document information required by an application unit. These
content instructions may be provided by the application
program. Alternatively, the content instructions can be input-
ted via an input device such as a keyboard, a touch screen, 10
a mouse, a notepad, a voice recognition device, or the like.
The computer 230 is also used to format selected stored
document information into the transmission format used by
an application unit based on transmission format instruc-
tions. The transmission format instructions may be provided 15
by the application program.. Alternatively, the transmission
format instructions can be inputted via a keyboard, a touch
screen, a mouse, a notepad, a voice recognition device, or
the like.
6
ASCII format. The scanner memory 220 can be. for
example. an electronic, magnetic, or optical memory.
FIG. 3A illustrates an enlarged view of the computer 230
of FIG. 1. This view will be used to describe a second mode
of operation. In this second mode of operation, the hard copy
document 100 is scanned and a copy of the document 100 is
displayed on display 232 of computer 230. based on the
contents of information temporarily stored in scanner
memory 220. After the document is displayed on display
232. the computer 230 interactively prompts the user to
identify the location of specific pieces of information on the
hard copy document. In the FIG. 3A illustration. this prompt
message is indicated as the message beginning with the
arrow.
For example, the prompt message can ask the user to
Thus, the computer 230 is used to generate an input file
for a particular application unit. The computer 230 is con-
nected to scanner memory 220, main, or permanent,
memory 250, a printer 260, and application units 270, via a
bus 240. Although FIG. 1 illustrates use of a bus to connect
components together, it is understood that any routing or
connecting link. implemented in hardware or software or
both, can be employed instead of, or in addition to, a bus.
Instructions to or in the computer 230 control the main
memory 250, the printer 260, the application units 270, and
the bus 240. Instructions to or in computer 230 can also
control exchanges of information with scanner memory 220.
identify the location of account number information on the
hard copy document. The user then uses an input device,
such as keyboard 234 or mouse 236 or a touch screen,
notepad, voice recognition device. or other input device to
20 position a cursor on the display to identify the location of the
information requested by the prompt message. For example.
the cursor could be used to define a block (which could be
highlighted) containing the requested information. followed
by a mouse "enter" click. In this example. the user would
25 move the mouse to identify the location of the account
number information contained on the hard copy document
100. The computer 230 then stores the information which
has been identified by the user as account number informa-
tion in the appropriate address or subfile or as the appropri-
When the computer 230 generates an input file for a
particular document, the computer 230 can send this input
file directly to an application unit or can store this input file
in the main memory 250 until required by an application
unit. The main memory 250 may also optionally store a copy
of the image information for the hard copy document and the
textual information for the hard copy document. Thus, the
image information and textual information from the hard
copy document can be retrieved and printed out on printer
260. In addition, image and textual information stored in
scanner memory 220 or in main memory 250 can be used to
form additional input files at the time of input or at a later
time, based on content instructions and transmission format
instructions. Thus, the invention can, at the discretion of the
user. eliminate the need to retain copies of hard copy
documents, permitting a paperless office.
The application units 270 include particular application
programs and devices which are controlled in accordance
with information contained on hard copy document 100.
30 ate variable or parameter. or data field. in memory. The
computer then prompts the user to identify the location of
other information on the hard copy document. such as.
statement date information. The process proceeds until all of
the desired information has been stored into the appropriate
35 locations in memory.
FIG. 3B illustrates a variation of the second mode for
interactively prompting the user for information. In FIG. 3B,
the display is split into two portions. A left-hand portion
40 232L displays the image of the hard copy document and a
right-hand portion 232R displays the required application
program information. For example. in FIG. 3B. portion
232R displays a spreadsheet used by an application pro-
gram. While observing the split display. the user can input
45 instructions to associate specific pieces of information on the
hard copy document (for example. the vendor name indi-
cated by the mouse arrow 232A) with particular subfiles in
memory (for example. the vendor field next to which the
cursor 232C appears), using a mouse or other input device( s)
50 or both. The split display also allows the user to generate
content format instructions while observing the information
required for a particular application program on the right-
hand portion.
100 which contains information to be processed by the
instant invention. The document illustrated in FIG. 2 is a bill
from XYZ Corporation to customer ABC Corporation. FIG. 55
2 is only an example of a type of document that can be
processed by the instant invention.
These second modes of operation are efficient for small
businesses which receive a small number of a wide variety
of invoices. since the user does not necessarily have to store
all of the information that appears on the hard copy docu-
ment. A further advantage is that data input is quicker. easier.
and more accurate than with previous keyboard methodol-
In a first operational mbde, the scanner 210 stores all of
the information extracted off of hard copy document 100 in
the scanner memory 220 or, alternatively, in main memory 60
250. The extracted information is stored in two forms. The
actual image of the hard copy document 100 is stored as
image information in the scanner memory 220. In addition,
the scanner memory 220 stores textual information recog-
nized on the hard copy document 100 by. for example, 65
employing standard character recognition software. In the
preferred embodiment. the textual information is stored in
ogy. In addition, by specifying the location on the hard copy
document of information, the user may optionally create a
template, to be described in further detail below. for each
different type of invoice. This template is stored for future
use when another hard copy document in the same format is
received.
More specifically. instructions from computer 230 can
direct the scanner 210 and scanner memory 220, and/or main
Exhibit 2
56
5,768,416
7
memory 250, to scan andlor store only specific portions of
hard copy document 100. After the interactive prompts
required to obtain information for a desired application
program, the unused information stored in scanner memory
220 or 250 can be erased. Further. scanning of a second 5
identical document can be limited to only those portions of
the document which contain needed information.
More specifically, in FIG. 2, the lines 10 drawn around
certain portions of the document represent the areas which
the user has previously identified as the portions of a 10
document to be extracted by the scanner 210 and stored in
scanner memory 220 andlor main memory 250. Since the
logo 20 and the message 30 have not been identified as an
area to be scanned and stored, these areas are not scanned
and stored in subsequent documents. Since the user has 15
previously associated each of the areas 10 with a specific
subfile of information. e.g., the account number, the scanned
information is stored in memory locations corresponding to
that subf lie.
8
The information processing module 2.0 is the module
which coordinates and drives the entire system. In the
preferred embodiment. the information processing module
2.0 is implemented primarily by computer 230.
FIG. 7 illustrates information processing data flow in the
maintain library module 2.1. The maintain library module
2.1 maintains a library of irnaae information. for example, a
digitized image representing the actual appearance of the
hard copy document, and textual information of the hard
copy documents for reference during processing. This
library can be incorporated within scanner memory 220,
main memory 250, or another independent memory. for
example, a RAM disk. The maintain library module 2.1
includes a store document module 2.1.1, a correct errors
module 2.1.2 .. a retrieve document module 2.1.3, and a
document file 2.1.4. These modules operate collectively to
store. retrieve, and correct document information.
Data Processing
FIGS. 4-10 illustrate the flow of data in the FIG. 1
preferred embodiment. FIG. 4 illustrates the overall data
flow for the FIG. 1 preferred embodiment. The preferred
embodiment includes an input process module 1.0. an infor-
mation processing module 2.0. and an output processing
module 3.0. The information processing module 2.0 is
equipped to receive instructions from and transmit informa-
tion to a user. The information processing module 2.0 can
also transmit to and receive information from a remote
external device through communication interface 4.0. Input
process module 1.0 and output processing module 3.0 can
also access communication interface 4.0. A module is imple-
mented in hardware. software, or a combination of hardware
and software. The specific implementation for a particular
business application depends upon a variety of factors, for
example, the relative costs of hardware and software imple-
mented systems, the frequency with which a user will want
The store document module 2.1.1, prior to routing the
document to the document file 2.1.4, may provide informa-
20 tion on recognition errors which may have occurred while
inputting the document For example, the store document
module 2.1.1 identifies that a character contained on hard
copy document 100 was not recognized. The store document
module 2.1.1 also optionally causes a copy of the document
25 and its parsing to be displayed on the display 232 for
confirmation by the user. The user may utilize this oppor-
tunity to identify any errors in the displayed document and.
in conjunction with the correct errors module 2.1.2, to revise
the document's parsing. if necessary, prior to storage of the
30 document in memory. The module 2.1.1 also provides a
facility for the user to name a particular hard copy document
for cataloging. storage. and retrieval purposes. After the
document is named, the store document module 2.1.1 stores
copies of the document in the document file 2.1.4.
35
to expand or modify the system. and the like.
FIG. 5 is a more detailed diagram of the input process
module 1.0 of FIG. 4. The input process module 1.0 includes
40
a character input module 1.1, an image input module 1.2.
and. in the preferred embodiment, a character recognition
device 1.3. The character input module inputs textual 45
information. such as alphanumeric characters. from an input
device such as keyboard 234. The image input module 1.2
inputs image information. for example, a digitized image of
the actual appearance of hard copy document 100. Textual
information can include textual input from an input device 50
such as keyboard 234 and textual information extracted from
the document by character recognition device 1.3. Both
types of information comprise an input document which is
transmitted to information processing module 2.0. In the
FIG. 1 preferred embodiment. the processing performed by 55
input process module 1.0 occurs in scanner memory 220.
computer 230. and main memory 250.
The correct errors module 2.1.2 processes instructions
from the user to correct errors identified by the store
document module 2.1.1 and errors that have been spotted by
the user during the confirmation process.
The retrieve document module 2.1.3 permits the user to
retrieve a copy of a document previously stored in the
document file 2.1.4. As described above, long-term storage
is provided by main memory 250. if necessary.
data flow diagram for the maintain definitions module 2.2 of
FIG. 6. The maintain definitions module 2.2 allows the user
to define system and document parameters and maintains the
definitions of these system and document parameters. The
maintain definitions module 2.2 includes a define template
module 2.2.1 which allows the user to specify the location
of information on the document. This information provided
by the user defines a template which is used to extract
information off the document and to associate the extracted
information with a particular variable or subflle. These
templates are illustrated by boxes 10 in the FIG. 2 example
of a hard copy document. The maintain definitions module
2.2 can also access templates previously defined by the user
and stored in main memory 250. Templates can also be
provided as part of software packages developed by program
FIG. 6 illustrates information processing data flow for the
FIG. 1 preferred embodiment. that is, FIG. 6 illustrates data
flow in the information processing module 2.0. 60 developers.
The information processing module 2.0 includes a main-
tain library module 2.1. to be described in further detail
below in conjunction with FIG. 7, a maintain definitions
module 2.2. to be described in further detail below in
conjunction with FIG. 8. and a process document module 2.3 65
to be described in further detail below in conjunction with
FIG. 9.
relationships module 2.2.2. The define relationships module
2.2.2 allows the user to define data relationships. or logical
relationships, between pieces of information extracted from
the hard copy document. These pieces of infonnation are
then used to generate an input file for a selected computer
application unit. The user defines these relationships by
Exhibit 2
57
5,768,416
9
content instructions. Alternatively. content instructions to
define relationships can be provided by application software.
If the user provides these content instructions, the content
instructions are inputted via keyboard 234 or via another
input device such as a notepad, a voice recognition device, 5
or the like. Examples of content instructions. data. and
logical relationships will be described in further detail in
conjunction with FIGS. 11 and 12A, IlB, and UC.
format module 2.2.3. The define format module 2.2.3 allows 10
the user to define transmission formats for an input file
which is then transmitted to a selected computer application
unit. Selection of the transmission format of the. input file is
accomplished by the user through use of transmission format
instructions. Alternatively. the applications software itself
can generate its own transmission format instructions. When 15
the user must specify transmission format instructions. the
transmission format instructions are inputted via keyboard
234 or via another input device such as a notepad. a voice
recognition device. or the like. A further description of
various transmission formats will be provided below in 20
conjunction with FIGS. 12A. 12B, 12C. 13A. 13B. and 13C.
10
The user enters the system by providing instructions to the
information processing module 2.0. The user then instructs
the information processing module 2.0 to conduct maintain
library processing. maintain definitions processing. or pro-
cess document processing.
If the user selects maintain ltbrary processing. the user
then provides instructions to maintain or modify the docu-
ment library through the maintain library module 2.1. For
example. the user can direct the inputting and storage of a
hard copy document 100 or can retrieve and output a
document. The user requests inputting of a document
through the store document module 2.1.1. The system then
prompts the user to specify a storage location for the
inputted document. The document is then read in by the
input process module 1.0. A textual copy and/or an image
copy are stored into the document file 2.1.4. Errors which
have occurred during inputting are identified and corrected
by the correct errors module 2.1.2 and the user. The correc-
tions are reflected in the document information stored in
document file 2.1.4.
The retrieve document module 2.1.3 is used to retrieve
and output a document. The system prompts the user to
specify the storage location of a document and the type of
document copy. for example. a textual or an image copy. to
be outputted. The document is then outputted by the output
process module 3.0.
If the user initially selected maintain definitions
processing. the user would instruct the system to maintain
A select definitions module 2.2.4 is also included in the
maintain definitions module 2.2. The select definitions mod-
ule 2.2.4 allows the user to store and select a set of
definitions to be used for processing the document. The 25
definitions identify pieces of information on the document
by. for example. absolute location. variable location. or
relative location. or by proximity to key words and/or
symbols. These definitions are described in further detail
below by way of an illustrative example.
data flow diagram for the process document module 2.3. The
process document is module 2.3 processes the document
after the document has been stored in the system. The
process document module 2.3 gathers the appropriate infor-
mation which has been stored. and creates input file(s) 2.3.3 35
for the selected application unit. The process document
module 2.3 then transmits the inputfile(s) via bus 240 and/or
communication interface 4.0 to an application unit 270. an
output device such as printer 260. or to main memory 250.
30 and/or modify parameter definitions through the maintain
definitions module 2.2. For example. the user can define and
maintain a document template for extracting selected por-
tions of information off of the hard copy document. The user
can use the template to extract selected portions of infor-
The process document module 2.3 includes an extract 40
data module 2.3.1. This module extracts data off of the
document in accordance with the user's instructions. for
example. the user-defined template. or through the interac-
tive mode.
The process document module 2.3 also includes a preap- 45
plication process module 2.3.2 which gathers and associates
information extracted from the document in accordance with
content instructions. This module prompts the user for any
additional information required to satisfy the relationships
defined by the content instructions. The preapplication pro- 50
cess module 2.3.2 also places the selected information into
the Transmission format defined by the transmission format
instructions.
The preapplication process module 2.3.2 also generates
the input file 2.3.3 for the selected application in accordance 55
with the appropriate instructions. The input file 2.3.3 is then
transmitted to bus 240 and/or communication interface 4.0
for transmission to a particular application unit 270.
mation off of the hard copy document when the document is
originally inputted. or alternatively. the user can use the
template to identify selected portions of information for
extraction off of an image copy of the document. In creating
the template. the user identifies pieces of information on the
document to be extracted and assigns a variable name. or
subf ile. to each piece of data.
The location of data to be extracted can be defined in a
number of ways other than by use of a template. For
example. the user can designate the absolute location of
information on the document with respect to a grid overlaid
on the document. e.g . always on line 3. starting in column
1. The user can also identify information by specifying the
relative location of information to be extracted. e.g .. always
two lines below the piece of data named "salutation".
starting in column 3. The user can also specify the location
of information to be extracted by variable location specifi-
cation. For example. if the hard copy document is a letter.
the module would conduct a key word search for the term
"Dear Sir:". Wherever this term "Dear Sir:" is located. this
piece of data wbuld be associated with the variable specified
by the user. for example. the variable "salutation." In
addition. a defined set of conventional symbols can be used
to signify certain recurring data items for the convenience of
users of the instant invention. For example. a "@" symbol
can be used to delineate the vendor name as follows:
"@XYZ Corporation@". Other examples of the use of
symbols to delineate information will be described with
reference to FIG. 14.
FIG. 10 illustrates a detailed output data flow diagram for
output module 3.0. Output module 3.0 outputs a textual 60
and/or image copy of the document. In the FIG. 1 preferred
embodiment. output module 3.0 is implemented by printer
260. associated software. and associated interface circuitry.
The maintains definition module 2.2 is also used to
65 maintain data relationships in accordance with content
instructions and to maintain input file formats in accordance
with transmission format instructions. Relationships are
Operation
Examples of operation of a preferred embodiment will
now be described.
Exhibit 2
58
5,768,416
11 12
defined and maintained between pieces of data. specified by. into the computer 230 rather than by sending the input file
for example, the names of variables. through the define to a remote application unit 270.
relationships module 2.2.2. The names of pieces of data on An illustrative example of the processing described above
the document are retrieved by. for example, the define will now be described.
template module 2.2.1. and are passed to the define rela- 5 The user inputs instructions via keyboard 234 or another
tionships module 2.2.2. The user may then provide any input device which indicate that the user desires to input and
additional pieces of data needed to generate an input file for store a document. The computer 230 then prompts the user
a particular application program or unit. such as an input file for the name of the document. In this exampl.e, the user
line number. The user, the applications software, andlor desires to input the document of FIG. 2 and therefore names
instructions previously stored in memory then establishes 10 the document "XYZ Corp. Bill 12101/86." The computer
the contents of the input file by defining relationships then prompts the user to feed the hard copy do(:ument 100
between pieces of data using content instructions. Specific into the scanner 210. The image of the hard copy document
examples of content instructions will be discussed below in is displayed on display .232. The computer then prompts the
conjunction with FIGS. 11. 12A. 12B. 12C, 13A, 13B. and user to identify the account number on the document. By use
13C. 15 of the mouse 236 or other input device to position a cursor
The user andlor the applications software defines and on the display. the user indicates the location of the account
maintains the transmission format of the input file to be used number. The account number is then read-in to a subfile
by a particular application program or unit through the named "Account Number."This process proceeds until all of
define format module 2.2.3 in accordance with transmission the desired information has been read-in and stored.
format instructions. This is accomplished by defining the 20 In this particular example, no errors were encountered
parameters to be used by the preapplication process module while inputting the document. The user then directs that the
2.3.2 in generating an input file. Parameters which would document be stored for future reference in a document file.
typically be required to generate an input file would include Some time later, the user desires to retrieve and output the
the character type. e.g., text or pixel; delimiters used document and to generate input files based on information
between pieces of data. e.g . a slash or a semicolon; end of 25 from the document. The computer 230 prompts the user for
line characters. e.g .. a carriage return or a line feed; and end the name of the document and the type of output. The user
of file characters. Examples of transmission formats will be responds with "XYZ Corp. Bill 12101/86" for a printed
described in further detail below in conjunction with FIGS. textual copy. The document is then retrieved from the
11. 12A. 12B. 12C, 13A. 13B. and 13C. document file and passed to the printer 260 for printing.
30
If the user initially selected process document processing. In order to generate an input file for a specific application
the interface will then proceed to process the document program. the user selects the option to define a document
through use of the process document module 2.3. For template for use when each month's XYZ Corporation bill
example. the user can extract specific portions of data from arrives. Accordingly, the user instructs the system to display
an image copy of a document. can generate an input file for 35 a copy of an XYZ Corporation bill on the display 232. The
transmission to an application program. or can directly user then identifies pieces of data by absolute locations. That
process information interactively with an application pro- is. the user assigns specific names to information located at
gram. specific portions of the document. In this example. the user
If the user desires to extract specific portions of data from would input the following information:
an image copy of a hard copy document which has already 40 Vendor-text. line 1. one line. column 1, 80 characters;
been stored in memory. the user uses the extract data module Account number-numeric, line 6. one line. column 25. 9
2.3.1 to identify a document to be processed. The document characters;
is then retrieved by the retrieve document module 2.1.3 and Statement date-date. line 9, one line, column 25, 8 char-
passed to the extract data module 2.3.1. The user can also
acters;
select parameter definitions through the select definitions 45 Payment date-date. line 11. one line. column 25. 8 char-
module 2.2.4.
The selected document template or parameter definition is
passed to the extract data module 2.3.1. The extract data
module 2.3.1 extracts pieces of data from the image copy of
the document. as defined by the document template defini- 50
tion or the parameter definitions or both. This document data
is then passed to preapplication process module 2.3.2.
The interface generates input file(s) 2.3.3 by use of the
preapplication process module 2.3.2. The selected data rela-
tionship definition. as defined by the content instructions, 55
and the selected record format definitions, as defined by the
transmission fonnat instructions, are passed to the preappli-
cation process module 2.3.2. The preapplication process
module 2.3.2 assembles the input file in accordance with the
content instructions. The preapplication process module 60
2.3.2 also prompts the user for any additional pieces of data
which need to be provided by the user. The input file is
converted to the desired transmission format in accordance
with the transmission format instructions. This physically
formatted data is then stored in the input file 2.3.3.
65
The user can also use an application program to process
information by loading the particular application program
acters;
Previous balance-currency. line 7, one line, column 75. 9
characters;
New charges-currency, line 8. one line, column 75. 9
characters;
Other debits-currency, line 10. one line. column 75. 9
characters;
Finance charges-currency. line 12. one line. column 75. 9
characters;
Payments-currency, line 13. one line. column 75. 9 char-
acters;
Other credits-currency. line 14, one line, column 75. 9
characters;
New balance-currency. line 15. one line, column 75, 9
characters.
The user also identifies data with variable locations. In
this particular example, a variable location is specified as
follows:
Heading 2-line. value="Mail To:"
The identification of Heading 2 as line information means
that the system will search for occurrences of the character
Exhibit 2
59
5,768,416
13
string "Mail To:" and assign the line number which contains
this character string to Heading 2.
The user also identifies data by relative locations. In this
example. the user identifies the following relative location:
Mail To-text. Heading 2+1. 3 lines. column 60. 25 char- 5
acters per line.
The instructions above instruct the system to assign the
textual information beginning on one line after Heading 2
and continuing for 3 lines. in column 60. to the Mail To
subfile. 10
As an alternative to inputting the actual line. column. and
character numbers. the user can identify desired portions of
the document by blocking. or highlighting. the desired
portions using the mouse or other input device. In this case.
the computer converts the highlighted portions into corre- 15
sponding line. column. and character numbers.
FIG. 11 lists data corresponding to the hard copy docu-
ment of FIG. 2 and the associated variable or subfile names.
Next. the user desires to define data relationships in
accordance with content instructions. Examples of the type
of contents which can be specified by a user are illustrated 20
in FIGS. 12A. 12B. and 12C.
In this particular example . three separate departments of
ABC Corporation require information from the XYZ Cor-
poration bill. The first department requires vendor. account
number. statement date. payment date. previous balance. 25
new charges. debits. finance charges. payments. and new
balance information. The second and third departments
require mail to information and previous balance informa-
tion. Each of these departments have their own application
program which utilizes this information. 30
The user employs content instructions to designate how
pieces of information. which have been extracted off of hard
copy document 100. are directed to particular departments.
that is, particular application programs. FIG. 12A illustrates
the contents of the information to be transmitted to the first 35
department. FIG. 12B illustrates the information to be
transmitted to the second department. FIG. 12C illustrates
the information to be transmitted to the third department.
The content instructions. therefore. parse the information
shown in FIG. 11 to various application programs. as shown 40
by FIGS. 12A. 12B. and 12C. Content instructions can also
be used to identify additional pieces of data which are
required for the input files of the particular application
programs. In this particular example. the specific application
programs from the three departments all require numeric 45
record number information. numeric horizontal position
infonnation, numeric vertical position information. and date
received information. The horizontal and vertical position
infonnation is used by the application program to specify the
location of the received information on a spreadsheet appli- 50
cation program. in this example. The user may know in
advance the content format required by each application
program. that is. in this example. the location and type of
infonnation specified on the spreadsheet. The user may also
employ the split display mode described with reference to 55
FIG. 3B to generate content format instructions.
Using the content instructions. the user establishes the
following contents for the input file corresponding to FIG.
12A:
Record number, horizontal position. vertical position, 60
vendor;
Record number, horizontal position.
account number;
vertical position.
14
Record number. horizontal position, vertical position.
payment date;
Record number. horizontal position. vertical position.
previous balance;
new charges;
finance charges;
payments;
new balance.
Next. transmission format instructions are employed to
define the transmission fonnat of the input file for a specific
application program or unit. FIG. 13A illustrates the trans-
mission input file corresponding to FIG. 12A. FIG. 13B
illustrates the transmission input file corresponding to FIG.
12B. FIG. 13C illustrates the transmission input file corre-
sponding to FIG. 12C. A comparison of FIGS. 12B and 12C
reveals that FIGS. 12B and 12C have the same contents.
However. the information illustrated in FIG. 12B is being
sent to a different application program than the information
in FIG. 12C. These application programs require different
transmission input formats as illustrated in FIGS. 13B and
13C. More specifically. the application program that
receives the input file illustrated in FIG. 13B uses the greater
than sign as a delimiter whereas the application program
which receives the transmission input file shown in FIG.
13C uses a back-slash as the delimiter.
After the contents and the transmission format for the
input file have been defined. and any additional information
has been inputted. the input file is assembled and transmitted
to the particular application program.
FIG. 14 illustrates another example of a hard copy docu-
ment containing information to be processed by the instant
invention. The hard copy document illustrated in FIG. 14 is
first scanned and information from the hard copy document
is stored into a memory. The interface 200 then identifies
portions of the hard copy document corresponding to vari-
ous variables by recognizing a defined set of symbols. In the
FIG. 14 example. triangles delineate the mailing address.
circles delineate the statement date. and squares delineate
the new charges. Information from these portions of the hard
copy document is stored in the corresponding memory
locations or subfiles for each variable. The same set of
symbols can be used to identify the same information from
one document to the next. Thus, even if the physical formats
of documents are not fixed from one document to the next.
a diversity of hard copy documents can be processed without
manually inputting data by recognition of the defined sym-
bols.
Examples of readily available application programs are
Quicken and Lotus 1,2.3 both of which are widely utilized
in the business community. Quicken. for example is an
easy-to-utilize program for writing checks and preparing
business records. Payee, amount and address information
may readily be transmitted from scanner memory 220 and/or
main memory 250 to the Quicken application program for
check writing functions and ledger keeping purposes. Lotus
is a well known spreadsheet program which may process
data input into specified cells once this data is placed in
conventional Lotus format.
Record number. horizontal position,
statement date;
Record number. horizontal position.
date received;
vertical position.
vertical position.
Thus. the instant invention provides an integrated and
65 comprehensive system for handling information from a hard
copy document. thus permitting a paperless office. In
addition. the invention permits data. extracted off of a hard
Exhibit 2
60
5,768A16
15
copy document. to be easily manipulated into various logical
and transmission formats required by a particular application
unit. The invention also provides a low cost system for
inputting information from a wide variety of hard copy
documents into a memory.
The foregoing description has been set forth merely to
illustrate preferred embodiments of the invention and is not
intended to be limiting. Modifications are possible without
departing from the scope of the invention.
5
For example. letters. checks. forms. pictures. reports, 10
music scores. film, and other types of hard copy documents
can be processed by the invention for accounts payable!
receivable accounting. inventory control. record keeping.
budgeting. data base management. music transcription.
forms processing. computerized art. survey and question- 15
naire processing. statistical data analysis. correspondence
processing and other applications.
Other automated digitizing units can be used in addition
to or as an alternative to use of the scanner 210 as an input
unit. Any electrical. magnetic. or optical device which 20
extracts information off of a hard copy document. thereby
eliminating the need to manually input significant amounts
of information from the hard copy document is suitable for
use as an automated digitizing unit. In addition, information
can be input by user responses and digital and analog signals 25
generated from various devices. and from computer files
from other computer systems. Suitable hardware for input-
ting data includes a keyboard. a light pen, a mouse, a touch
screen. a laser scanner. a microphone. a tablet. a disk drive,
a magnetic tape drive. and a modem.
30
The interface 200 can also output information in forms
other than a hard copy of textual or image information. For
example. the interface 200 can output system responses.
computer files. and digital and analog signals for transmis-
sion to other computer systems or to control systems. 35
Suitable hardware for outputting information includes a disk
drive. a magnetic tape drive. a cathode ray tube, a plasma
screen. a printer. a plotter, a film developer. an amplifier, and
a modem.
Since modifications of the described embodiments incor- 40
porating the spirit and substance of the invention may occur
to persons skilled in the art, the scope of the invention should
16
(b)(I) scanning a hard copy document and displaying
an image of at least a portion of the hard copy
document on a display;
(b)(2) interactively receiving instructions from a user to
create a template which identifies at least one loca-
tion on said image as containing information to be
converted to text data corresponding to a field of
information required by an application program; and
(b)(3) storing said template;
(c) inputting information from a hard copy document
according to an automatic mode which includes the
following substeps
(c)(l) scanning a hard copy document to generate
document information; and
(c)(2) matching at least a portion of said document
information with a template created to transfer con-
verted text data to at least one field of information
required by an application program.
2. A method as set forth in claim 1. further comprising the
steps of:
defining a set of symbols which designate fields of
detecting the presence of a particular one of said defined
set of symbols on a hard copy document and extracting
a field of information required by an application pro-
gram based on said detecting.
step of:
displaying a split-screen image showing at least a portion
of said hard copy document and at least one field of
information required by an application program during
the interactive mode.
step of:
identifying said located information in step (a)(2) by
blocking portions of said image on said display using
an input device.
S. A method as set forth in claim 3, further comprising the
step of:
identifying said located information in step (a)(2) by
blocking portions of said image on said display using
an input device.
be limited solely with respect to the appended claims and
equivalents.
What is claimed is:
45 step of:
the interactive template creation mode.
1. A method of inputting information from a diversity of
hard copy documents to a computer, comprising at least one
of the following steps (a). (b). and (c):
(a) inputting information from a hard copy document
according to an interactive mode which includes the 50
following substeps
(a)(I) scanning a hard copy document and displaying a
digitized image of at least a portion of the hard copy
(a)(2) interactively receiving instructions from a user 55
for identifying information on said digitized image
and converting at least said identified information to
converted text data corresponding to at least one field
of information required by an application program;
and
(a)(3) transferring said converted text data to said field
of information;
60
(b) creating a template to enable transfer of converted text
data corresponding to at least one field of information
required by an application program according to an 65
interactive template creation mode which includes the
following substeps
step of:
creating a template in step (b )(2) by blocking portions of
said image on said display using an input device.
S. A method as set forth in claim 6, further comprising the
step of:
creating a template in step (b)(2) by blocking portions of
step of:
storing a digitized image of a least a portion of a hard copy
document in an image memory.
10. A method as set forth in claim 1, further comprising
the step of:
storing text data from the hard copy document in a text
memory.
11. A method as set forth in claim 1. further comprising
the step of:
Exhibit 2
61
5,768,416
17
storing a digitized image of a least a portion of a hard copy
document in an image memory; and
memory.
12. A method as set forth in claim 1, further comprising 5
the step of:
repeating substeps (a)(2) and (a)(3) at least once for other
identified information and another field of information.
the steps of repeating substeps (a)(2) and (a)(3) for another 10
different application program.
the steps of:
repeating substeps (a)(l), (a)(2) and (a)(3) for another
different application program; and 15
creating another template for another different application
program.
IS. A method as set forth in claim 1. further comprising
the step of:
repeating substeps (c)(l) and (c)(2) for another different 20
hard copy document.
16. A method as set forth in claim 1 wherein in step (b)(2)
said template identifies a plurality of locations on said image
on said display as containing a field of information required
by an application program. 25
the step of creating another template for another ditTerent
application program.
at least one of the steps of:
repeating substeps (a)(2) and (a)(3) for another different
application program; and
program.
30
the steps of repeating substeps (a)(l), (a)(2) and (a)(3) for
another different application program.
hard copy documents to a computer, comprising at least two
of the following steps (a), (b), and (c):
according to an interactive mode which includes the
following substeps
40
(aXl) scanning a hard copy document and displaying a
digitized image of at least a portion of the hard copy 45
(a)(2) interactively receiving instructions from a user
converted text data corresponding to at least one field 50
and
of information;
(b) creating a template to enable transfer of converted text 55
required by an application program according to an
following substeps
(b)(l) scanning a hard copy document and displaying 6(J
(b )(2) interactively receiving instructions from a user to
tion on said image as containing information to be 65
18
(b)(3) storing said template;
following substeps
(c)(l) scanning a hard copy document to generate
information with a template created to transfer con-
the steps of:
the step of:
the step of:
identifying said information in step (a)(2) by blocking
portions of said image on said display using an input
device.
the step of:
device.
the step of:
the step of:
the step of:
the step of:
storing a digitized image of at least a portion of a hard
copy document in an image memory.
the step of:
memory.
the step of:
copy document in an image memory; and
memory.
the step of:
repeating substeps (aX2) and (a)(3) at least once for other
Exhibit 2
62
5,768.416
19
the steps of repeating substeps (a)(2) and (a)(3) for another
different application program.
the steps of:
repeating substeps (a)(I), (a)(2) and (a)(3) for another
different application program; and
5
program.
the step of:
repeating substeps (c)(I) and (c)(2) for another different
hard copy document
35. A method as set forth in claim 20, wherein in step 15
(b )(2) said template identifies a plurality of locations on said
image on said display as containing a field of information
the step of creating another template for another different 20
the steps of:
repeating substeps (a)(2) and (a)(3) for another different
program.
the steps of repeating substeps (a)(I), (a)(2) and (a)(3) for
another different application program.
at least two of the steps of:
(a) repeating substeps (a)(I), (a)(2) and (a)(3) at least once
for another different hard copy document;
25
30
(b) repeating substeps (b)(I), (b)(2) and (b)(3) at least 35
once for another different hard copy document; and
(c) repeating substeps (c)( 1) and (c )(2) for another dif-
ferent hard copy document.
40. A method as recited in claim 39 wherein said at least 40
two step are steps (a) and (b).
41. A method as recited in claim 39 wherein said at least
two step are steps (a) and (c).
42. A method as recited in claim 39 wherein said at least
two step are steps (b) and (c). 45
hard copy documents to a computer, comprising each of the
following steps (a), (b), and (c):
20
(b)(I) scanning a hard copy document and displaying
(b)(2) interactively receiving instructions from a user to
tion on said image as containing infomJation to be
(b )(3) storing said template;
following substeps
(c)(I) scanning a hard copy document to generate
information with a template created to .mnsfer con-
the steps of:
the step of:
the step of:
device.
the step of:
device.
the step of:
according to an interactive mode which includes the
following substeps
50 49. A method as set forth in claim 43, further comprising
the step of:
(a)(I) scanning a hard copy document and displaying a
digitized image of at least a portion of the hard copy
(a)(2) interactively receiving instructions from a user
converted text data corresponding to at least one field
so. A method as set forth in claim 48, further comprising
55 the step of:
and
of information;
60
(b) creating a template to enable transfer of converted text
required by an application program according to an 65
following substeps
the step of:
copy document in an image memory.
the step of:
memory.
the step of:
Exhibit 2
63
5,768,416
21
copy document in an image memory; and
memory.
54. A method as set forth in claim 43. further comprising 5
the step of:
repeating substeps (a)(2) and (a)(3) at least once for other
the steps of repeating substeps (a)(2) and (a)(3) for another 10
di1Ierent application program.
the step of:
repeating substeps (a)(l). (a)(2) and (a)(3) for another 15
di1Ierent application program; and
creating another template for another di1Ierent application
program.
the step of: 20
repeating substeps (c)(l) and (c)(2) for another di1Ierent
hard copy document.
58. A method as set forth in claim 43. wherein in step
(b X2) said template identifies a plurality of locations on said
image on said display as containing a field of information 25
the step of creating another template for another di1Ierent
60. A method as set forth in claim 43. further comprising 30
the steps of:
repeating substeps (a)(2) and (a)(3) for another di1Ierent
22
creating another template for another di1Ierent application
program.
61. A method as set forth in claim 1. 20 or 43 further
comprising at least one of the steps of:
(a) repeating substeps (a)(l). (a)(2) and (a)(3) at least once
for another di1Ierent hard copy document;
(b) repeating substeps (b)(l). (b)(2) and (b)(3) at least
once for another di1Ierent hard copy document; and
(c) repeating substeps (c)(l) and (c)(2) for another dif-
ferent hard copy document.
62. A method as set forth in claim 61 wherein said at least
one step is step (a).
one step is step (b).
one step is step (c).
the steps of repeating substeps (aXl). (a)(2) and (a)(3) for
another di1Ierent application program.
the steps of:
repeating sub steps (a)(l). (a)(2) and (a)(3) at least once
for another different hard copy document;
repeating substeps (b)(l). (b)(2) and (bX3) at least once
for another di1Ierent hard copy document; and
repeating substeps (c)(l) and (c)(2) for another different
hard copy document.
* * * * *
Exhibit 2
64
JS 44 (Rev. 09/11)
CIVIL COVER SHEET
The JS 44 civil cover sheet and the information contained herein neither replace nor supplement the filing and service of pleadings or other papers as required by law, except as provided
by local rules of court. This form, approved by the Judicial Conference of the United States in September 1974, is required for the use of the Clerk of Court for the purpose of initiating
the civil docket sheet. (SEE INSTRUCTIONS ON NEXT PAGE OF THIS FORM.)
I. (a) PLAINTIFFS DEFENDANTS
(b) County of Residence of First Listed Plaintiff County of Residence of First Listed Defendant
(EXCEPT IN U.S. PLAINTIFF CASES) (IN U.S. PLAINTIFF CASES ONLY)
NOTE: IN LAND CONDEMNATION CASES, USE THE LOCATION OF
THE TRACT OF LAND INVOLVED.

(c) Attorneys (Firm Name, Address, and Telephone Number) Attorneys (If Known)
II. BASIS OF JURISDICTION (Place an X in One Box Only) III. CITIZENSHIP OF PRINCIPAL PARTIES (Place an X in One Box for Plaintiff)
(For Diversity Cases Only) and One Box for Defendant)
1 U.S. Government 3 Federal Question PTF DEF PTF DEF
Plaintiff (U.S. Government Not a Party) Citizen of This State 1 1 Incorporated or Principal Place 4 4
of Business In This State
2 U.S. Government 4 Diversity Citizen of Another State 2 2 Incorporated and Principal Place 5 5
Defendant (Indicate Citizenship of Parties in Item III) of Business In Another State
Citizen or Subject of a 3 3 Foreign Nation 6 6
Foreign Country
IV. NATURE OF SUIT (Place an X in One Box Only)
CONTRACT TORTS FORFEITURE/PENALTY BANKRUPTCY OTHER STATUTES
110 Insurance PERSONAL INJURY PERSONAL INJURY 625 Drug Related Seizure 422 Appeal 28 USC 158 375 False Claims Act
120 Marine 310 Airplane 365 Personal Injury - of Property 21 USC 881 423 Withdrawal 400 State Reapportionment
130 Miller Act 315 Airplane Product Product Liability 690 Other 28 USC 157 410 Antitrust
140 Negotiable Instrument Liability 367 Health Care/ 430 Banks and Banking
150 Recovery of Overpayment 320 Assault, Libel & Pharmaceutical PROPERTY RIGHTS 450 Commerce
& Enforcement of Judgment Slander Personal Injury 820 Copyrights 460 Deportation
151 Medicare Act 330 Federal Employers Product Liability 830 Patent 470 Racketeer Influenced and
152 Recovery of Defaulted Liability 368 Asbestos Personal 840 Trademark Corrupt Organizations
Student Loans 340 Marine Injury Product 480 Consumer Credit
(Excl. Veterans) 345 Marine Product Liability LABOR SOCIAL SECURITY 490 Cable/Sat TV
153 Recovery of Overpayment Liability PERSONAL PROPERTY 710 Fair Labor Standards 861 HIA (1395ff) 850 Securities/Commodities/
of Veterans Benefits 350 Motor Vehicle 370 Other Fraud Act 862 Black Lung (923) Exchange
160 Stockholders Suits 355 Motor Vehicle 371 Truth in Lending 720 Labor/Mgmt. Relations 863 DIWC/DIWW (405(g)) 890 Other Statutory Actions
190 Other Contract Product Liability 380 Other Personal 740 Railway Labor Act 864 SSID Title XVI 891 Agricultural Acts
195 Contract Product Liability 360 Other Personal Property Damage 751 Family and Medical 865 RSI (405(g)) 893 Environmental Matters
196 Franchise Injury 385 Property Damage Leave Act 895 Freedom of Information
362 Personal Injury - Product Liability 790 Other Labor Litigation Act
Med. Malpractice 791 Empl. Ret. Inc. 896 Arbitration
REAL PROPERTY CIVIL RIGHTS PRISONER PETITIONS Security Act FEDERAL TAX SUITS 899 Administrative Procedure
210 Land Condemnation 440 Other Civil Rights 510 Motions to Vacate 870 Taxes (U.S. Plaintiff Act/Review or Appeal of
220 Foreclosure 441 Voting Sentence or Defendant) Agency Decision
230 Rent Lease & Ejectment 442 Employment Habeas Corpus: 871 IRSThird Party 950 Constitutionality of
240 Torts to Land 443 Housing/ 530 General 26 USC 7609 State Statutes
245 Tort Product Liability Accommodations 535 Death Penalty IMMIGRATION
290 All Other Real Property 445 Amer. w/Disabilities - 540 Mandamus & Other 462 Naturalization Application
Employment 550 Civil Rights 463 Habeas Corpus -
446 Amer. w/Disabilities - 555 Prison Condition Alien Detainee
Other 560 Civil Detainee - (Prisoner Petition)
448 Education Conditions of 465 Other Immigration
Confinement Actions
V. ORIGIN
Transferred from
another district
(specify)
(Place an X in One Box Only)
1 Original
Proceeding
2 Removed from
State Court
3 Remanded from
Appellate Court
4 Reinstated or
Reopened
5 6 Multidistrict
Litigation
VI. CAUSE OF ACTION
Cite the U.S. Civil Statute under which you are filing (Do not cite jurisdictional statutes unless diversity):

Brief description of cause:
VII. REQUESTED IN
COMPLAINT:
CHECK IF THIS IS A CLASS ACTION
UNDER F.R.C.P. 23
DEMAND $ CHECK YES only if demanded in complaint:
JURY DEMAND: Yes No
VIII. RELATED CASE(S)
IF ANY
(See instructions):
JUDGE DOCKET NUMBER
DATE SIGNATURE OF ATTORNEY OF RECORD
FOR OFFICE USE ONLY
RECEIPT # AMOUNT APPLYING IFP JUDGE MAG. JUDGE
CONTENT EXTRACTION AND TRANSMISSION LLC
Out of state
Zimmerman and Weiser, Anatoly S. Weiser, Esq., 3525 Del Mar Heights
Road, #295, San Diego, CA 92130; (858) 720-9431
SAN DIEGO COUNTY CREDIT UNION
San Diego
35 USC 271
Patent Infringement
07/27/2012 /s/ Anatoly S. Weiser
Print Save As... Reset
'12CV1878 RBB BTM
JS 44 Reverse (Rev. 09/11)
INSTRUCTIONS FOR ATTORNEYS COMPLETING CIVIL COVER SHEET FORM JS 44
Authority For Civil Cover Sheet
The JS 44 civil cover sheet and the information contained herein neither replaces nor supplements the filings and service of pleading or other papers as required
by law, except as provided by local rules of court. This form, approved by the Judicial Conference of the United States in September 1974, is required for the
use of the Clerk of Court for the purpose of initiating the civil docket sheet. Consequently, a civil cover sheet is submitted to the Clerk of Court for each civil
complaint filed. The attorney filing a case should complete the form as follows:
I. (a) Plaintiffs-Defendants. Enter names (last, first, middle initial) of plaintiff and defendant. If the plaintiff or defendant is a government agency, use only
the full name or standard abbreviations. If the plaintiff or defendant is an official within a government agency, identify first the agency and then the official, giving
both name and title.
(b) County of Residence. For each civil case filed, except U.S. plaintiff cases, enter the name of the county where the first listed plaintiff resides at the
time of filing. In U.S. plaintiff cases, enter the name of the county in which the first listed defendant resides at the time of filing. (NOTE: In land condemnation
cases, the county of residence of the defendant is the location of the tract of land involved.)
(c) Attorneys. Enter the firm name, address, telephone number, and attorney of record. If there are several attorneys, list them on an attachment, noting
in this section (see attachment).
II. Jurisdiction. The basis of jurisdiction is set forth under Rule 8(a), F.R.C.P., which requires that jurisdictions be shown in pleadings. Place an X in one
of the boxes. If there is more than one basis of jurisdiction, precedence is given in the order shown below.
United States plaintiff. (1) Jurisdiction based on 28 U.S.C. 1345 and 1348. Suits by agencies and officers of the United States are included here.
United States defendant. (2) When the plaintiff is suing the United States, its officers or agencies, place an X in this box.
Federal question. (3) This refers to suits under 28 U.S.C. 1331, where jurisdiction arises under the Constitution of the United States, an amendment to the
Constitution, an act of Congress or a treaty of the United States. In cases where the U.S. is a party, the U.S. plaintiff or defendant code takes precedence, and box
1 or 2 should be marked.
Diversity of citizenship. (4) This refers to suits under 28 U.S.C. 1332, where parties are citizens of different states. When Box 4 is checked, the citizenship of
the different parties must be checked. (See Section III below; federal question actions take precedence over diversity cases.)
III. Residence (citizenship) of Principal Parties. This section of the JS 44 is to be completed if diversity of citizenship was indicated above. Mark this section
for each principal party.
IV. Nature of Suit. Place an X in the appropriate box. If the nature of suit cannot be determined, be sure the cause of action, in Section VI below, is
sufficient to enable the deputy clerk or the statistical clerks in the Administrative Office to determine the nature of suit. If the cause fits more than one nature of
suit, select the most definitive.
V. Origin. Place an X in one of the seven boxes.
Original Proceedings. (1) Cases which originate in the United States district courts.
Removed from State Court. (2) Proceedings initiated in state courts may be removed to the district courts under Title 28 U.S.C., Section 1441. When the petition
for removal is granted, check this box.
Remanded from Appellate Court. (3) Check this box for cases remanded to the district court for further action. Use the date of remand as the filing date.
Reinstated or Reopened. (4) Check this box for cases reinstated or reopened in the district court. Use the reopening date as the filing date.
Transferred from Another District. (5) For cases transferred under Title 28 U.S.C. Section 1404(a). Do not use this for within district transfers or multidistrict
litigation transfers.
Multidistrict Litigation. (6) Check this box when a multidistrict case is transferred into the district under authority of Title 28 U.S.C. Section 1407. When this
box is checked, do not check (5) above.
Appeal to District Judge from Magistrate Judgment. (7) Check this box for an appeal from a magistrate judges decision.
VI. Cause of Action. Report the civil statute directly related to the cause of action and give a brief description of the cause. Do not cite jurisdictional statutes
unless diversity. Example: U.S. Civil Statute: 47 USC 553
Brief Description: Unauthorized reception of cable service
VII. Requested in Complaint. Class Action. Place an X in this box if you are filing a class action under Rule 23, F.R.Cv.P.
Demand. In this space enter the dollar amount (in thousands of dollars) being demanded or indicate other demand such as a preliminary injunction.
Jury Demand. Check the appropriate box to indicate whether or not a jury is being demanded.
VIII. Related Cases. This section of the JS 44 is used to reference related pending cases if any. If there are related pending cases, insert the docket numbers
and the corresponding judge names for such cases.
Date and Attorney Signature. Date and sign the civil cover sheet.

Content Extraction and Transmission v. San Diego County Credit Union

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Content Extraction and Transmission v. San Diego County Credit Union

Uploaded by

Copyright:

Available Formats

1

You might also like