Index of /~anoop/distrib/trigen

Icon  Name                    Last modified      Size  Description
[DIR] Parent Directory                             -   
[   ] austen-sents.py         25-Oct-2007 22:50  207   
[   ] austen.lm               21-Oct-2010 15:47  8.0M  
[   ] austen.lm.bz2           07-Sep-2010 23:19  2.1M  
[TXT] austen.txt              21-Oct-2010 15:46  1.3M  
[TXT] austen.txt.bz2          07-Sep-2010 23:17  486K  
[   ] austen_heldout.py       10-Jan-2011 20:38  175   
[TXT] austen_heldout.txt      15-Sep-2010 11:09  676K  
[TXT] austen_samples.txt      10-Jan-2011 20:38  1.8K  
[   ] austen_sents.py         10-Jan-2011 20:38  219   
[   ] austen_train.py         10-Jan-2011 20:38  199   
[TXT] austen_train.txt        15-Sep-2010 11:08  1.3M  
[   ] create-lm.sh            10-Jan-2011 20:38  251   
[   ] create_model            10-Jan-2011 20:38  534   
[   ] entropy_demo.py         10-Jan-2011 20:38  2.8K  
[   ] equi.model              10-Jan-2011 20:38   30   
[   ] equigen.pl              10-Jan-2011 20:38  371   
[   ] evalequi.pl             10-Jan-2011 20:38  313   
[   ] evaltrigen.pl           10-Jan-2011 20:38  555   
[   ] evaluni.pl              10-Jan-2011 20:38  312   
[TXT] gen-from-lm.py          26-Oct-2007 20:11  3.7K  
[TXT] gen_from_bigram.py      10-Jan-2011 20:38  3.3K  
[TXT] gen_from_lm.py          10-Jan-2011 20:38  3.7K  
[TXT] gen_from_trigram.py     10-Jan-2011 20:38  3.7K  
[TXT] gen_from_unigram.py     10-Jan-2011 20:38  3.5K  
[   ] gmon.out                15-Sep-2010 11:09  595K  
[   ] pretty.pl               10-Jan-2011 20:38  179   
[TXT] readme.txt              10-Jan-2011 20:38  3.8K  
[   ] show                    10-Jan-2011 20:38   32   
[   ] tri.model.bz2           10-Dec-2000 12:08  1.0M  
[   ] tri.pl                  10-Jan-2011 20:38  955   
[TXT] trigen.pl               10-Jan-2011 20:38  2.0K  
[   ] trigen.tar.gz           10-Dec-2000 12:08  1.0M  
[   ] trigram_generator.pl    10-Jan-2011 20:38  2.4K  
[   ] tripl.pl                10-Jan-2011 20:38  202   
[   ] uni.model               10-Jan-2011 20:38   36   
[   ] unigen.pl               10-Jan-2011 20:38  594   


Implementation of the Shannon game. Programs for the following cases:

generation of letters based on equiprobable unigrams:

equigen.pl equi.model 200
evalequi.pl equi.model 200

generation of characters based on unigram prob:

unigen.pl uni.model 200
evaluni.pl uni.model 200

generation of words based on trigrams trained from the WSJ:

trigen.pl tri.model.bz2 200 | pretty.pl 

evaltrigen.pl tri.model.bz2 200 > outfile
./show outfile 

To produce a trigram model from a corpus:

./create_model corpus

This creates 'corpus.model'

Read:

A Mathematical Theory of Communication. C. E. Shannon. The Bell System
Technical Journal. Vol. 27, pp. 379--423, 623--656, July, October,
1948.

Here are some examples of the program output:

From corpus.model (WSJ text):

In short , the Dalai Lama admitted the obvious: `` China as a lesser-known company that Nippon is trying to sell securities in the week that demand had slowed for the investments in leveraged buy-out that Drexel is `` pleased '' by C.S. Godshalk , a diversified portfolio for as they can trumpet other themes: toughness on crime .

Mr. Banks on his transition today when he died .

'' Schumann 's `` Ping Pong Polka , '' he said , encourages a `` standards of public ridicule .

They lost the White House chief of staff , according to Kyodo News Service , covering 198 public and private enterprise , with shavings of white truffle , costs that are even more to meet today with its plan to do so .

From corpus2.model (assorted poems):


Waters fall 
Creation looking on

The world does not encourage a perfectly rational army because
when we die - 
I want life to go on aching still

Through centuries above 
Enlightened to a larger pain 
By contrast with
the dew 
The frost beheads it at its play 
In
accidental power 
The bone that has no price 
The walls began to tell ;

Creation

--END--

The moon was but a play 
The only one
forestalling mine 
His eccentricities 
You do n't know where it goes 
Could be
the smile 
Upon her friend she could bestow

Were such her silver will ! 
And
night is darkening round me so

I live on the run 
Because 
I do n't have a missing friend -- 
And 
I cannot explain that in lines

--END--


Moon

They say that ' time assuages,
' -- 
Time never did assuage; 
An actual suffering strengthens,

As sinews do, with age 
Time is a movable feast

First the smiles, then the letting go 
The
research
forms take final
shape,

Each figure
like a page 
Of prancing poetry 
This world is mud -
luscious the little children of the iron bell 
Calls
the faithful
father

--END--

It
wasted
from
my sight, must remain
a delight 
Nor the heavy knitting of the past 
Yes, it
makes a happy sound

It
was a sentence-generating organism
and 
Wilhelm

Reich who said he was a sentence-generating organism and 
Wilhelm

Reich who said he
was a bundle
of conditioned reflexes and 
Chomsky who said that man was absurd

--END--