Xtreams

'Don't cross the streams!'

Michael Lucas-Smith
Martin Kobetic

Cincom Smalltalk Engineering

Smalltalk Solutions 2011
 Status
Trunk: Stable
Documentation: http://code.google.com/p/xtreams
License: MIT
Project started: November 30th 2007

Platforms
VisualWorks
Squeak and Pharo (Nicolas Cellier)
Gemstone (Dale Henrichs)
Slate (Brian Rice)

... thanks to all who have provided feedback

and patches
 Why?
simplification and consistency
Incomplete vs atEnd
positioning and 'surprising' buffers
read and write are different

composition
no deep class hierarchy any more

capabilities
transformations
substreams

scalability
chunks vs elements
recycling
 Structure
core
Defines the API and Core Classes.
Support package is for portability.

terminals
Unique to each platform, wraps the
source and destination objects in to streams.

transforms
collecting, rejecting, selecting...
character encoding
binary transformations
object marshaling
 Structure
substreams / parsing
limiting, element or subcollection ending
slicing and stitching
parsing expression grammars
multiplexing (experimental)

xtras
compression
cryptography
chunking

applications
SSH2 (Xtreams-SSH2)
IRC (Xtreams-IRC)
EXIF Metadata (Xtreams-Photo)
 reading / writing
collections
'example' reading.
OrderedCollection new writing.
Transcript writing.

(RingBuffer new: 4) writing.

SharedQueue new writing.
 reading / writing
externals
'example.txt' asFilename reading.
OS.Stdin reading.
(External.CIntegerType unsignedChar
gcMalloc: 50) writing.
(OS.SocketAccessor
newTCPclientToHost: 'localhost'
port: 1234) reading.

transcendentals
[0] reading. /dev/zero
[:x | ] writing. /dev/null
Kernel.ObjectMemory reading.
Random new reading.
 API
common
Incomplete.
stream terminal.
stream close.
stream isReadable; isWritable; isPositionable.

reading
input get.
input read: 5.
input read: 5 into: collection.
input read: 5 into: collection at: 2.
input rest.

do:, do:separatedBy:, select:, reject:, collect:, fold:,

detect:(ifNone:), inject:into:, groupedBy:
 API
writing
output put: 'example'.
output write: $x.
output write: input.
output write: 'example'.
output write: 4 from: input.
output write: 4 from: 'example'.
output write: 3 from: 'example' at: 4.
output conclusion.

output insert: 'example'.

output insert: 4 from: 'example'.
output insert: 3 from: 'example' at: 4.

output print: #( 1 2 3 ).
output cr; bell; q; qq; space; space: 2; tab; tab: 3
 API
seeking
positionable := stream positioning.
transform a non-positionable stream in to
a positionable stream

stream ++ 200.
seek forward from where we are

stream -- 200.
seek backward from where we are

stream += 200.
seek forward from the start of the stream

stream -= 0. (skip to end)

seek backward from the end of the stream
 API
seeking
stream explore: [stream read: 2].
seek within the block, but return to where
we were once done

stream position. stream position: 4.

change the position of the stream

stream available.
elements left to consume on the stream
(not to be mistaken with data
left in socket buffer)

stream length.
elements in the stream
 QA
Questions?
 Transforms
Collection Style
collecting:, selecting:, injecting:into:, doing:, ...

Specialized Transforms
encoding:, encodingBase64, encodingHex,
compressing, en/decrypting:key:iv:, hashing:
interpreting:, marshaling

General Transforms
transforming: [ :in :out | out put: in get ]
 Collection Transforms
random := Random new reading.
random := random collecting: [ :f | (f * 256) floor ].
random contentsSpecies: ByteArray.
random read: 10.

current := 0.
integers := [current := current + 1] reading.
integers read: 10.
even := integers selecting: [ :i | i even ].
even read: 10.
 Character Encoding
input := 'xtreams.cha' asFilename reading.
input := input encoding: #utf8.
input read: 50.
input close.

(#[13 10 10 13] reading encoding: #ascii) rest.

(ByteArray new writing encoding: #ascii)
cr; conclusion
 Composition / Stacking
ones := [ 1 ] reading.
twoAndUp := ones
injecting: 1
into: [ :previous :one | previous + one ].

sieve := OrderedCollection new.

primes := twoAndUp rejecting: [ :i |
(sieve anySatisfy: [ :p | i \\ p = 0 ])
ifTrue: [ true ]
ifFalse: [ sieve add: i. false ] ].
primes read: 10.
 Morse Code
Message
... -- .- .-.. .-.. - .- .-.. -.-

DecodingTree
- << * >> .
T E
M N A I
O G K D W R U S
Q Z Y C X B J P L F V H

($ ($T ($M ($O) ($G ($Q) ($Z)))

($N ($K ($Y) ($C)) ($D ($X) ($B)))
($E ($A ($W ($J) ($P)) ($R () ($L)))
($I ($U () ($F)) ($S ($V) ($H)))
 Morse Code Decoding
($ ($T ($M ($O) ($G ($Q) ($Z)))
($N ($K ($Y) ($C)) ($D ($X) ($B)))
($E ($A ($W ($J) ($P)) ($R () ($L)))
($I ($U () ($F)) ($S ($V) ($H)))

('... -- .- .-.. .-.. - .- .-.. -.- ' reading

transforming: [ :in :out || node beep |
node := MorseTree.
[ beep := in get.
beep = Character space
] whileFalse: [
node := beep = $.
ifTrue: [ node at: 3 ]
ifFalse: [ node at: 2 ] ].
out put: node first ]
) rest
 Morse Code Encoding
(String new writing
transforming: [ :in :out |
out write: (Morse at: in get);
space ]
) write: 'SMALLTALK';
conclusion
 QA
Questions?
 Substreams
* sentences in text
* parts of messages in a protocol
* files in an archive

size
(Object comment reading limiting: 10) rest.

bounding criteria
(Object comment reading ending: $.) rest.
(Object comment reading ending: [:e | '.?!' includes:
(Object comment reading ending: ' is the') rest.

streams of substreams
slicing streams into substreams
stitching streams from substreams
 Substreams - limiting:
output := String new writing.
Number withAllSubclasses do: [ :class |
[ (output limiting: 40) write: class comment.
] on: Incomplete do: [ :ex | output -- 3; write: '...' ].
output cr ].
output conclusion.
 Substreams - ending:
output := String new writing.
Number withAllSubclasses do: [ :class |
[ (output ending: $. inclusive: true)
write: class comment
] on: Incomplete do: [].
output cr ].
output conclusion
 Slicing
reading
input := 'xtreams.cha' asFilename reading.
input := input encoding: #utf8.
[ lines := (input ending: Character cr) slicing.
(lines ++ 10000; get) rest
] ensure: [ input close ].

writing
output := String new writing.
blurbs := (output limiting: 40) slicing.
Number withAllSubclasses do: [ :class |
[ blurbs get write: class comment.
] on: Incomplete do: [ :ex | output -- 3; write: '...'
output cr ].
output conclusion.
 Slices of Slices
reading
input := 'aaa#bb#c##!1#22#33#444' reading.
messages := (input ending: $!) slicing.
parts := (messages get ending: $#) slicing.
parts collect: [ :p | p rest ].

writing
output := String new writing.
messages := (output closing: [ output put: $! ])
slicing.
#((aa bb cc dd ee) (xxx yy z)) do: [ :m |
message := messages get.
parts := (message closing: [ message put: $# ])
slicing.
m do: [ :p | parts get write: p ] ].
output conclusion
 Stitching Reads
((1 to: 5) reading, (6 to: 10) reading) rest.
((1 to: 10) reading limiting: 3) slicing stitching rest.

files := '..' asFilename reading.

[ | fn | fn := files get.
fn isDirectory ifTrue: [ files := fn reading, files ].
fn
] reading collect: [ :f | f asString ].

directories := ElasticBuffer new: 10 class: Array.

directories put: '..' asFilename.
[ directories get reading doing: [ :filename |
filename isDirectory ifTrue: [
directories put: filename]]
] reading stitching collect: [ :f | f asString ].
 Stitching Writes
output := ByteArray new writing.
buffer := RingBuffer on: (ByteArray new: 5).
[ (buffer writing limiting: buffer cacheSize)
closeBlock: [
output put: buffer readSize;
write: buffer ];
yourself
] reading stitching
write: (1 to: 12);
close.
output conclusion
 QA
Questions?
 Back to Binary
Stream --> Objects
Interpreting streams
Object marshaling
Parsing Expression Grammars
Erlang 'Bit Syntax'

Objects --> Stream

Interpreting streams
Object marshaling
 Interpreting
stream interpreting: #long.
stream interpreting: #signedLonglong_be.

(input interpreting: #float) read: 10.

(output interpreting: #double) put: Double pi.
 Marshaling
goals
Cross-Smalltalk (not achieved yet)
Space Efficiency
Speed Efficiency

to replace...
BOSS
Opentalk-STST
Parcels (sort of)
SIXX
 Marshaling
ObjectMarshaler
Separate from the Read/Write streams.
Pluggable marshaling strategy, for different protocols
Uses Pragmas to allow extensions with
a computed hash to detect different
versions.

Analysis
Read without instantiating objects to
diagnose a stream.
 Marshaling
output := ByteArray new writing marshaling.
output put: 100 asValue.

output conclusion =
#[83 84 83 84 20 4 21 199 91 7 32 29 82
111 111 116 46 83 109 97 108 108 116
97 108 107 46 85 73 46 86 97 108 117
101 72 111 108 100 101 114 0 1 28 22
100]
 Marshaling
(Xtreams.ObjectAnalyseStream on:
output conclusion reading) rest =
'0+10 header: #[83 84 83 84 20 4 105 117 236 8]
10+36 record
10+1 class id: 32
11+31 class description: UI.ValueHolder
42+1 object id: 1
43+3 object: UI.ValueHolder
43+1 dependents
43+1 class id: 28 Core.UndefinedObject
43+1 class id: 28
44+0 nil
44+2 value
44+2 class id: 22 Core.SmallInteger
44+1 class id: 22
45+1 byte integer: 100'
 Marshaling
output conclusion reading marshaling get.
a ValueHolder on: 100
 Parsing
Parsing Expression Grammars
Erlang 'bit syntax'
Reusable Grammars
Grammar composition
 Parsing
Parsing Expression Grammars

grammar :=
'Sentence <- Whitespace? (Word / Punctuation)*
Word <- [a-zA-Z'']+
Punctuation <- [,;:()]'.

sentenceParser := PEG.Parser parserPEG

parse: 'Grammar'
stream: grammar
actor: PEG.ParserParser new
 Parsing
Applications
Transformed Namespaced class names
of Xtreams during import/export via
Monticello using the Smalltalk PEG
grammar with an Actor that contained
a single method.

Implemented an IRC client where data

read from the client stream was parsed
with a grammar and interpreted using
a single method on the IRC client itself.
The whole IRC implementation is one
class.
 Parsing
Applications
Transformed Smalltalk, Javascript, XML,
CSS3 grammars in to a Javascript
lexer to perform syntax color highlighting
in javascript in the web browser for
WebVelocity 1.1

This presentation. The slides of this

presentation are built by parsing Smalltalk
methods comprised of wiki syntax in comments
and smalltalk code. The wiki grammar is used
with an actor to generate Text objects that
display what you see right now.
 Parsing
Parsing Expression Grammars
PEG: character streams only
Read one character: .
Read one of many characters: [a-zA-Z]
Read a sequence of characters: ''TITLE''

Moving beyond...
Grammar Composition
Compile grammar into bytecode
on lightweight class
Read one character: {character/utf8}
Read one byte: {integer/unsigned/little}:8
Read one of many characters in utf16:
[a-zA-Z]/utf16
Read a sequence of characters in utf8:
''TITLE''/utf8
 QA
Questions?
 Xtras
Various non-core transformations

Compression
calls ZLib

Cryptography
calls BCrypt (CNG) on Windows
calls OpenSSL's libcrypto everywhere else
hashing (MD5, SHA1, SHA256, SHA512, ...)
hashing with key (HMAC with MD5, SHA1, ...)
encryption (RC4, AES, DES, ...)

HTTP Chunking
 Xtras - Examples
hashing
(ObjectMemory imageFilename reading
hashing: 'MD5'
) -= 0; close; digest.

encryption
key := random read: 16.
((String new writing
encodingBase64
encrypting: 'RC4' key: key)
compressing
encoding: #utf8
) write: Object comment;
conclusion.
 Xtras - SSH2
does
transport
authentication: public-key, password
channel/session management
data transfer
channel requests: exec, env, exit-status
scp

does not
terminal session/emulation: pty-req, shell
port forwarding (tunnels)
scp atime/mtime
 Xtras - SSH2
keys := SSH2KeysTest sampleKeys.
config := SSH2Configuration new keys: keys.
server := ('-thishost-' asIPv4: 2222) listen accepting.
[ (SSH2ServerConnection on: server get)
configuration: config;
when: SSH2Announcement
do: [ :m | Transcript cr; print: m ];
accept;
waitForDisconnect;
close
] ensure: [ server close. keys release ]
 Xtras - SSH2
home := '$(HOME)' asLogicalFileSpecification asFilenam
user := home tail.
keys := SSH2Keys fromUser: home.
config := SSH2Configuration new keys: keys.
client := ('localhost' asIPv4: 22) connect.
client := SSH2ClientConnection on: client.
client configuration: config.
client when: SSH2Announcement
do: [ :m | Transcript cr; print: m ].
[ service := client connect: user.
session := service session.
[ session exec: 'ls -l'
] ensure: [ session close ]
] ensure: [ client close. keys release ]
 QA
Questions?